Mutation Testing
Mutation testing introduces small artificial code changes to evaluate whether existing tests detect the resulting behavior differences.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/mutation-testing/
01What it means
Mutation testing creates program variants, or mutants, by changing operations such as comparisons or return values and reruns relevant tests. A mutant is killed when a test detects it and survives when the suite still passes. The method examines test sensitivity rather than simply whether code was executed; tool-specific scoring must distinguish surviving, uncovered, invalid, and equivalent cases.[1][2]
02Why it matters
Coverage can be high while assertions are too weak to notice wrong behavior. Surviving mutants direct attention to missing expectations, untested boundaries, and sometimes code that has no observable effect.
03When to use it
Use it selectively on important business rules or a test suite whose effectiveness is uncertain. Start with stable tests and a bounded code area. Running it across expensive end-to-end suites or chasing a perfect global score can consume more effort than the evidence justifies.
04How it works
- Confirm the baseline tests pass consistently.
- Select meaningful code and a suitable set of mutation operators.
- Generate mutants and execute the relevant tests.
- Inspect survivors to distinguish missing checks from equivalent behavior or tooling limits.
- Improve valuable assertions and rerun the affected mutations.
05A practical example
A fee calculator has full line coverage, but changing a threshold comparison from greater-than to greater-than-or-equal survives. The team adds an exact-boundary example based on the requirement. That mutant is then detected without adding unrelated tests.
06Where it breaks
Equivalent mutants may preserve behavior and cannot be killed by a correct test. Results depend on operators and tool rules, and execution cost can be substantial. A high mutation score cannot establish that requirements are correct or that integrations work.
Common mistakes
- Equating line coverage with mutation detection.
- Adding meaningless assertions just to improve the score.
- Treating every survivor as proof of a production defect.
07Sources
- PIT Mutation TestingPIT · Official documentationPIT Mutation Testing
- Mutant states and metricsStryker Mutator · Official documentationMutant states and metrics