Skip to content

Guide

Confirmation checks: how to stop false downtime alerts

Many false downtime alerts come from a single failed request: a dropped packet, a container restarting, a load balancer shuffling traffic. Confirmation checks make a monitor ask again before it raises the alarm.

Updated 25 October 2026 · By the Upcertain team

What a confirmation check is

When a scheduled check fails, the monitor does not open an incident straight away. It checks again after a short delay, and again, up to the number of confirmation checks you set. The incident opens only if every one of them fails too. If any of them passes, nothing happens: no incident, no alert.

In Upcertain each monitor has 0 to 5 confirmation checks, 15, 30, 60 or 120 seconds apart. New monitors start with one confirmation check after 15 seconds, which filters out single dropped requests while adding very little delay. Heartbeat monitors have none, because a missed deadline is already a confirmed failure.

The trade-off: certainty against speed

Every confirmation check adds its delay to the time before anyone hears about a real outage. Roughly, a failure that starts at a random moment is noticed by the next scheduled check, half an interval later on average, and then confirmed after the re-checks:

IntervalConfirmationAverage time to alertSlowestShorter failures ignored
30 snone15 s30 snone
30 s1 check after 15 s30 s45 sunder about 15 s
60 s1 check after 15 s45 s1 m 15 sunder about 15 s
30 s3 checks, 30 s apart1 m 45 s2 munder about 90 s
60 s3 checks, 30 s apart2 m2 m 30 sunder about 90 s
5 min3 checks, 60 s apart5 m 30 s8 munder about 3 min

The times ignore how long each request takes; a check that runs into its timeout adds that timeout too. The last column is the point of the exercise: a failure has to outlast the confirmation window to become an incident.

How to choose

  1. Start with the shortest failure you want to hear about. If a 30-second blip on your marketing site does not matter but a 2-minute outage does, the confirmation window should sit between the two.
  2. Measure your known noise. Deploys, nightly restarts and failovers produce failures of a predictable length. Set the window just longer than they last. The guide to false alarms during deploys works through an example.
  3. Keep the interval short and confirm instead. A 30-second interval with a 90-second confirmation window still notices a real outage within about two minutes. A 5-minute interval without confirmation is slower and still fires on blips.
  4. Different services, different settings. A payment API may deserve one quick re-check; a batch-processing dashboard can wait for three.

What confirmation does not replace

  • Maintenance windows for planned work. Time inside a window counts as maintenance, so a planned migration never opens an incident at all.
  • Alert delays per channel. Upcertain can delay one channel, say email to managers, by several minutes while Slack is told at once.
  • Fixing the cause. If a service fails for 60 seconds on every deploy, its users feel those 60 seconds. Confirmation keeps the pager quiet; zero-downtime deploys keep the users happy.

Setting it in Upcertain

Open a monitor, then Advanced settings, and set the number of confirmation checks and the time between them. To change many monitors at once, select them in the monitor list and use Confirmation. Heartbeat monitors are skipped.

Why not just check from several places?

Tools that check from many regions confirm by asking a second location at the same moment, which filters out network trouble between one checker and your server. Confirmation over time filters out short failures on your side, such as restarts and deploys, which several locations would all see at once. Upcertain checks from one region (Seoul), so it confirms over time.

Questions

Do failed checks that were not confirmed count as downtime?
They do not open an incident and no one is alerted. The individual failed check is still recorded in the monitor's check log.
What is a sensible default?
One confirmation check after 15 seconds for most websites and APIs. Raise it for services with known restarts or deploys that fail for longer.