Updated 25 October 2026 · By the Upcertain team
Why deploys fail health checks
- The old process stops before the new one is ready. A single-instance app restarts, and requests in between get a 502 from the proxy or a connection refused.
- The load balancer is slower than the app. New instances are marked healthy only after a few passing health checks, and old ones are drained, so for a moment there is nothing healthy behind the balancer: 503.
- Warm-up. The new process answers, but the first requests time out while caches fill, connections open or code compiles.
- Migrations. A schema change locks a table and requests that touch it fail until it finishes.
All of these end on their own, usually within a minute or two, and they repeat on every deploy. That pattern is what to design the monitoring around.
A worked example
Suppose a service returns 503 for between 45 and 90 seconds on each deploy, several times a day, and is checked every 30 seconds. With no confirmation, nearly every deploy opens an incident and posts to Slack. With one confirmation check after 15 seconds, most deploys still do, because the failure lasts far longer than 15 seconds.
Set three confirmation checks 30 seconds apart and the failure has to last about 90 seconds before an incident opens. Deploys that recover within that window pass silently; a real outage still alerts within about two minutes of starting. If deploys sometimes run longer, four checks (two minutes) gives more margin at the cost of thirty more seconds before a real outage is reported.
| Setting | Deploy noise (45 to 90 s of 503) | Real outage alerted after |
|---|---|---|
| No confirmation | Alert on almost every deploy | about 15 to 30 s |
| 1 check after 15 s | Alert on almost every deploy | about 30 to 45 s |
| 3 checks, 30 s apart | Deploys up to about 90 s stay quiet | about 2 min |
| 4 checks, 30 s apart | Deploys up to about 2 min stay quiet | about 2.5 min |
Fixes on the monitoring side
- Measure the deploy failure. Look at a few deploys in the monitor's check log and note how long the failures last.
- Set the confirmation window just above it. See confirmation checks for the arithmetic.
- Use maintenance windows for long, planned work such as a database upgrade. Time inside the window is shown as maintenance, not downtime.
- Delay slow channels, not fast ones. Keep the on-call Slack channel immediate and delay email to a wider group by a few minutes.
- Do not slow the interval to hide noise. A 5-minute interval misses most deploy failures by luck, but it also takes up to 5 minutes to notice a real outage and still alerts when a check lands in a deploy.
Fixes on the application side
- Run at least two instances and replace them one at a time.
- Add a readiness check so the load balancer sends traffic only to instances that can serve it, and drain connections before stopping old ones.
- Point the uptime monitor at a page that exercises the real path (or a
/healthendpoint that checks the database), not at a static file a CDN would serve anyway. - Run long migrations in steps that do not lock hot tables.
The monitoring fixes keep your team's attention for real problems. The application fixes remove the minute of errors your users still see during each deploy. Do both.