Incident Management
Incident management is the coordinated practice of detecting, containing, communicating, and learning from service disruptions.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/incident-management/
01What it means
Incident management provides a repeatable structure for handling events that materially degrade a service or create operational risk. It establishes severity levels, decision roles, communication channels, escalation paths, documentation, and transitions from active response to recovery and learning.[1][2]
- 01Detect + declareSeverity and activation criteria
- 02Assign rolesCommand, operations, communication, record
- 03Assess + mitigateImpact, priorities, containment
- 04Recover + monitorConfirm service and watch regression
- 05Learn + follow throughPostmortem and owned actions
Clear roles and a timestamped record preserve decision quality while responders work under pressure.
02Why it matters
During a disruption, uncertainty and coordination costs can compound technical failure. A known command structure helps responders form a shared picture, protect attention, make reversible decisions, and communicate honestly. It also preserves a timeline that supports later analysis instead of relying on stressed memories.
03When to use it
Activate the process when impact or uncertainty exceeds normal ticket handling, including outages, severe latency, data integrity threats, and critical dependency failures. Scale the structure to the event: a small incident may need one lead and a scribe, while a broad outage needs dedicated operations, communications, and planning roles.
04How it works
- Detect and declare the incident using clear severity and activation criteria.
- Assign an incident commander and owners for operations, communications, and documentation.
- Assess impact, establish priorities, and choose containment or mitigation actions.
- Send regular audience-appropriate updates and maintain a timestamped record.
- Confirm recovery, monitor for regression, hand off follow-up work, and schedule a postmortem.
05A practical example
A marketplace sees checkout failures across two regions. The on-call engineer declares a high-severity incident, an incident commander coordinates work, and a communications lead posts customer updates. Responders disable a new payment route, verify recovery using user-centered indicators, and capture decisions for the postmortem.
06Where it breaks
A process cannot compensate for missing telemetry, unsafe architecture, or exhausted responders. Too many roles can slow a minor event, while vague authority causes duplication. Fast restoration may require temporary mitigations that introduce debt, so follow-up ownership remains necessary after the channel quiets.
Common mistakes
- Letting the most senior person take command without explicit role assignment.
- Mixing investigation, stakeholder updates, and decision tracking in one chaotic stream.
- Closing the incident when graphs recover without verifying user impact or follow-up ownership.
07Sources
- Incident ResponseGoogle Site Reliability Engineering · Institutional practice chapterIncident Response
- Incident Management GuideGoogle Site Reliability Engineering · Operational guideIncident Management Guide