practiceEngineering & architecture

Chaos Engineering

Chaos engineering uses controlled experiments to test whether a system remains acceptable when realistic disruptive conditions occur.

Download Markdown (.md)

01What it means

Chaos engineering is the disciplined practice of forming a hypothesis about steady behavior, introducing a bounded disturbance, and observing whether the system preserves that behavior. Experiments may involve unavailable dependencies, delayed networks, resource pressure, or instance loss. The goal is evidence about resilience, not random breakage.[1][2]

Bounded disturbance against a steady-state hypothesis Inject a specific failure to learn whether user-relevant behavior survives
  1. 01Define steady stateMeasurable user or consumer behavior
  2. 02State hypothesisExpected behavior under one failure
  3. 03Bound experimentTarget, duration, blast radius, abort
  4. 04Introduce disturbanceDependency, network, resource, instance
  5. 05Learn + hardenAnalyze surprises, improve, repeat

Chaos engineering is disciplined resilience evidence, not random breakage. Abort thresholds, limited scope, and an immediate stop mechanism protect the system.

02Why it matters

Architecture diagrams and failover plans express intent, but production systems often contain hidden coupling, stale assumptions, and untested recovery paths. Carefully designed experiments reveal these weaknesses before an uncontrolled event does. They also train teams to observe, stop, and recover systems under realistic conditions.

03When to use it

Use chaos experiments after basic monitoring, incident response, and safety controls exist. Begin in a representative non-production environment or with a tiny production scope, then expand as confidence grows. Choose conditions suggested by past incidents, dependency risks, disaster scenarios, or important resilience claims.

04How it works

  1. Define measurable steady-state behavior from the user or consumer perspective.
  2. State a hypothesis about behavior during a specific failure condition.
  3. Limit scope with target selection, duration, abort thresholds, and an immediate stop mechanism.
  4. Run the smallest useful experiment while monitoring impact and system response.
  5. Analyze surprises, improve resilience, and repeat to confirm that changes work.

05A practical example

Example

A streaming platform hypothesizes that losing one recommendation-service zone will not materially increase playback-start failures. During a low-risk window, it disables a small set of instances, watches service-level indicators and queue depth, and aborts automatically if failure thresholds are crossed. Unexpected retry amplification leads to a client backoff fix.

06Where it breaks

Experiments can cause real harm, especially with poor scoping, weak observability, or unrecognized shared dependencies. Passing one test does not prove general resilience, and exercises can become theatrical if hypotheses are vague. Security, compliance, customer impact, and responder workload must shape the design.

Common mistakes

  • Injecting failure without a hypothesis or user-centered measure.
  • Starting with a broad production blast radius before validating controls.
  • Celebrating survival while failing to document and address surprising behavior.

07Sources

  1. Principles of Chaos EngineeringChaos Engineering Community · Foundational principlesPrinciples of Chaos Engineering
  2. What Is AWS Fault Injection Service?Amazon Web Services · Cloud service documentationWhat Is AWS Fault Injection Service?
View saved topics

Start typing to search titles, definitions, categories, and frameworks.