# Incident Management

Incident management is the coordinated practice of detecting, containing, communicating, and learning from service disruptions.

- Type: practice
- Domain: Engineering & architecture
- Level: Intermediate
- Last reviewed: 2026-09-22
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/incident-management/>)

## What it means

Incident management provides a repeatable structure for handling events that materially degrade a service or create operational risk. It establishes severity levels, decision roles, communication channels, escalation paths, documentation, and transitions from active response to recovery and learning.

## Why it matters

During a disruption, uncertainty and coordination costs can compound technical failure. A known command structure helps responders form a shared picture, protect attention, make reversible decisions, and communicate honestly. It also preserves a timeline that supports later analysis instead of relying on stressed memories.

## When to use it

Activate the process when impact or uncertainty exceeds normal ticket handling, including outages, severe latency, data integrity threats, and critical dependency failures. Scale the structure to the event: a small incident may need one lead and a scribe, while a broad outage needs dedicated operations, communications, and planning roles.

## How it works

1. Detect and declare the incident using clear severity and activation criteria.
2. Assign an incident commander and owners for operations, communications, and documentation.
3. Assess impact, establish priorities, and choose containment or mitigation actions.
4. Send regular audience-appropriate updates and maintain a timestamped record.
5. Confirm recovery, monitor for regression, hand off follow-up work, and schedule a postmortem.

## A practical example

A marketplace sees checkout failures across two regions. The on-call engineer declares a high-severity incident, an incident commander coordinates work, and a communications lead posts customer updates. Responders disable a new payment route, verify recovery using user-centered indicators, and capture decisions for the postmortem.

## Where it breaks

A process cannot compensate for missing telemetry, unsafe architecture, or exhausted responders. Too many roles can slow a minor event, while vague authority causes duplication. Fast restoration may require temporary mitigations that introduce debt, so follow-up ownership remains necessary after the channel quiets.

## Common mistakes

- Letting the most senior person take command without explicit role assignment.
- Mixing investigation, stakeholder updates, and decision tracking in one chaotic stream.
- Closing the incident when graphs recover without verifying user impact or follow-up ownership.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/incident-management/#visualization-incident-management>)

## Sources

1. [Incident Response](<https://sre.google/workbook/incident-response/>) — Google Site Reliability Engineering · Institutional practice chapter
2. [Incident Management Guide](<https://sre.google/resources/practices-and-processes/incident-management-guide/>) — Google Site Reliability Engineering · Operational guide

## Related topics

- [Blameless Postmortem](<https://wiki.soroush.uk/topic/blameless-postmortem/>)
- [Observability](<https://wiki.soroush.uk/topic/observability/>)
- [Disaster Recovery](<https://wiki.soroush.uk/topic/disaster-recovery/>)
