Updated 25 October 2026 · By the Upcertain team
Before anything breaks
- Agree who responds when. Even without a formal rotation, write down who covers evenings and weekends this week.
- Route alerts to people who can act. Each service's alerts go to its owners' channel; managers can get a delayed email instead.
- Cut the noise first. A team that gets false alarms stops reading alerts. Confirm failures before alerting; see confirmation checks.
- Have a status page ready, with its address in your docs and support replies.
- Keep a short note per service: where it runs, how to restart it, who to call at the provider.
When the alert fires
- Acknowledge. It tells everyone that someone is on it and stops repeat alerts. In Upcertain the alert links straight to the incident.
- Check the scope. Is it one service or many? Several monitors failing at once usually means a shared dependency.
- Tell customers if they are affected. A short status update within 10 to 15 minutes, even if you do not know the cause yet.
- Fix or mitigate. Roll back the last deploy first if it is a likely cause; investigate after service is restored.
- Write notes as you go in the incident: what you tried and what changed. You will not remember later.
Who does what
| Role | Small-team version |
|---|---|
| Responder | Whoever acknowledged; works on the fix |
| Communicator | A second person, if there is one, who posts updates and answers support |
| Decision-maker | The responder, unless they ask for help |
Splitting fixing from communicating is the one role change that helps even a two-person team: the person fixing should not also be writing customer updates.
After it is resolved
- Post a closing update on the status page.
- Within a few days, write a short review: what happened, how long, what users saw, what made it worse or better, and one or two changes.
- Make the changes small enough to actually do: an extra monitor, a better alert, a runbook line.
Measure only what you will use
Two numbers are enough to start: how long until someone acknowledged, and how long until it was resolved. Upcertain records both on every incident, and exports them as CSV. See incidents.
When to add more process
Add a formal on-call rotation and paging when outages outside working hours start costing real money, or when the same person keeps getting woken. Until then, clear ownership, quiet alerts and an honest status page cover most of what matters.
Questions
Do we need a formal post-mortem for every incident?
No. Write one for incidents customers noticed or that took longer than an hour. For the rest, a line in the incident's notes is enough.
How quickly should we update customers?
Within 10 to 15 minutes of confirming customers are affected, then at the interval you promise in that first message.