# Alerting

Alerting turns selected system signals into actionable notifications with a clear owner and response urgency.

- Type: practice
- Domain: Engineering & architecture
- Level: Intermediate
- Last reviewed: 2026-10-19
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/alerting/>)

## What it means

Operational alerting evaluates telemetry against rules and routes notifications when a condition needs attention. A page requests immediate intervention; a ticket can schedule nonurgent work. Alerting is narrower than observability, which supports investigation, and differs from an interface alert shown to a customer.

## Why it matters

Timely detection can reduce user harm, but repeated unactionable notifications exhaust responders. Good alerts explain why action is needed and connect the signal to an accountable response.

## When to use it

Use paging for urgent, actionable threats to user outcomes or reliability objectives. Route slower issues to tickets or dashboards. A signal that is interesting but requires no response generally should not wake someone.

## How it works

1. Define the user impact and the action an alert is meant to trigger.
2. Select signals and windows that distinguish sustained problems from noise.
3. Set ownership, severity, grouping, and escalation routes.
4. Include useful context and a tested response procedure.
5. Exercise delivery and review missed incidents, false alarms, and stale rules.

## A practical example

A payments service pages when failed payments consume its error budget unusually quickly across short and longer windows. A gradual disk-capacity trend creates a maintenance ticket instead. Responders can distinguish an immediate customer problem from planned capacity work.

## Where it breaks

No threshold perfectly balances detection speed and false alarms. Low traffic, delayed telemetry, and broken monitoring can obscure incidents. Alert routing and the monitoring system itself need checks; adding more rules is not a substitute for reliable signals.

## Common mistakes

- Paging on every metric threshold with no defined action.
- Leaving alerts without a current owner or escalation path.
- Ignoring failures in the notification pipeline itself.

## Sources

1. [Alerting on SLOs](<https://sre.google/workbook/alerting-on-slos/>) — Google SRE · Engineering handbook
2. [Alerting](<https://prometheus.io/docs/practices/alerting/>) — Prometheus · Official documentation

## Related topics

- [Observability](<https://wiki.soroush.uk/topic/observability/>)
- [Incident Management](<https://wiki.soroush.uk/topic/incident-management/>)
- [Error Budget](<https://wiki.soroush.uk/topic/error-budget/>)
