# Chaos Engineering

Chaos engineering uses controlled experiments to test whether a system remains acceptable when realistic disruptive conditions occur.

- Type: practice
- Domain: Engineering & architecture
- Level: Advanced
- Last reviewed: 2026-09-22
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/chaos-engineering/>)

## What it means

Chaos engineering is the disciplined practice of forming a hypothesis about steady behavior, introducing a bounded disturbance, and observing whether the system preserves that behavior. Experiments may involve unavailable dependencies, delayed networks, resource pressure, or instance loss. The goal is evidence about resilience, not random breakage.

## Why it matters

Architecture diagrams and failover plans express intent, but production systems often contain hidden coupling, stale assumptions, and untested recovery paths. Carefully designed experiments reveal these weaknesses before an uncontrolled event does. They also train teams to observe, stop, and recover systems under realistic conditions.

## When to use it

Use chaos experiments after basic monitoring, incident response, and safety controls exist. Begin in a representative non-production environment or with a tiny production scope, then expand as confidence grows. Choose conditions suggested by past incidents, dependency risks, disaster scenarios, or important resilience claims.

## How it works

1. Define measurable steady-state behavior from the user or consumer perspective.
2. State a hypothesis about behavior during a specific failure condition.
3. Limit scope with target selection, duration, abort thresholds, and an immediate stop mechanism.
4. Run the smallest useful experiment while monitoring impact and system response.
5. Analyze surprises, improve resilience, and repeat to confirm that changes work.

## A practical example

A streaming platform hypothesizes that losing one recommendation-service zone will not materially increase playback-start failures. During a low-risk window, it disables a small set of instances, watches service-level indicators and queue depth, and aborts automatically if failure thresholds are crossed. Unexpected retry amplification leads to a client backoff fix.

## Where it breaks

Experiments can cause real harm, especially with poor scoping, weak observability, or unrecognized shared dependencies. Passing one test does not prove general resilience, and exercises can become theatrical if hypotheses are vague. Security, compliance, customer impact, and responder workload must shape the design.

## Common mistakes

- Injecting failure without a hypothesis or user-centered measure.
- Starting with a broad production blast radius before validating controls.
- Celebrating survival while failing to document and address surprising behavior.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/chaos-engineering/#visualization-chaos-engineering>)

## Sources

1. [Principles of Chaos Engineering](<https://principlesofchaos.org/>) — Chaos Engineering Community · Foundational principles
2. [What Is AWS Fault Injection Service?](<https://docs.aws.amazon.com/fis/latest/userguide/what-is.html>) — Amazon Web Services · Cloud service documentation

## Related topics

- [Observability](<https://wiki.soroush.uk/topic/observability/>)
- [Incident Management](<https://wiki.soroush.uk/topic/incident-management/>)
- [Disaster Recovery](<https://wiki.soroush.uk/topic/disaster-recovery/>)
