Alerting
Alerting turns selected system signals into actionable notifications with a clear owner and response urgency.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/alerting/
01What it means
Operational alerting evaluates telemetry against rules and routes notifications when a condition needs attention. A page requests immediate intervention; a ticket can schedule nonurgent work. Alerting is narrower than observability, which supports investigation, and differs from an interface alert shown to a customer.[1][2]
02Why it matters
Timely detection can reduce user harm, but repeated unactionable notifications exhaust responders. Good alerts explain why action is needed and connect the signal to an accountable response.
03When to use it
Use paging for urgent, actionable threats to user outcomes or reliability objectives. Route slower issues to tickets or dashboards. A signal that is interesting but requires no response generally should not wake someone.
04How it works
- Define the user impact and the action an alert is meant to trigger.
- Select signals and windows that distinguish sustained problems from noise.
- Set ownership, severity, grouping, and escalation routes.
- Include useful context and a tested response procedure.
- Exercise delivery and review missed incidents, false alarms, and stale rules.
05A practical example
A payments service pages when failed payments consume its error budget unusually quickly across short and longer windows. A gradual disk-capacity trend creates a maintenance ticket instead. Responders can distinguish an immediate customer problem from planned capacity work.
06Where it breaks
No threshold perfectly balances detection speed and false alarms. Low traffic, delayed telemetry, and broken monitoring can obscure incidents. Alert routing and the monitoring system itself need checks; adding more rules is not a substitute for reliable signals.
Common mistakes
- Paging on every metric threshold with no defined action.
- Leaving alerts without a current owner or escalation path.
- Ignoring failures in the notification pipeline itself.
07Sources
- Alerting on SLOsGoogle SRE · Engineering handbookAlerting on SLOs
- AlertingPrometheus · Official documentationAlerting