# A/A Testing

A/A testing compares nominally identical experiment variants to check assignment, exposure, metrics, and analysis behavior before relying on a new experiment system.

- Type: method
- Domain: Data, experimentation & growth
- Level: Advanced
- Last reviewed: 2026-10-05
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/a-a-testing/>)

## What it means

An A/A test assigns eligible units to two conditions that should receive the same experience. It tests whether randomization, logging, metric computation, and statistical procedures behave as expected under no intended treatment difference.

## Why it matters

Experiment platforms can produce misleading results through assignment faults, event loss, and analysis errors. A/A tests make these failures observable before they distort product decisions.

## When to use it

Use A/A tests when launching or materially changing experimentation infrastructure, assignment, telemetry, or analysis methods. Do not use them as a substitute for ongoing quality checks.

## How it works

1. Define identical experiences and assignment.
2. Run the complete eligibility and analysis pipeline.
3. Check allocation, baseline metrics, missingness, and consistency.
4. Evaluate false-positive behavior against the documented method.
5. Repair anomalies before interpreting A/B results.

## A practical example

After an exposure-service migration, an A/A test finds a persistent mobile difference. The team discovers exposure is logged after a rendering failure and fixes the pipeline before a product test.

## Where it breaks

A passing A/A test cannot prove future experiments valid or expose all variant-specific defects. Small samples can miss problems, while many unplanned slices can create false alarms.

## Common mistakes

- Treating an A/A test as feature evidence.
- Ignoring a detected mismatch.
- Searching many segments until one appears significant.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/a-a-testing/#visualization-a-a-testing>)

## Sources

1. [Diagnosing Sample Ratio Mismatch in Online Controlled Experiments](<https://www.microsoft.com/en-us/research/publication/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonomy-and-rules-of-thumb-for-practitioners/>) — Microsoft Research · Research paper
2. [NIST/SEMATECH e-Handbook of Statistical Methods](<https://www.itl.nist.gov/div898/handbook/>) — National Institute of Standards and Technology · Government statistical reference

## Related topics

- [Sample Ratio Mismatch](<https://wiki.soroush.uk/topic/sample-ratio-mismatch/>)
- [Randomization](<https://wiki.soroush.uk/topic/randomization/>)
- [Instrumentation Plan](<https://wiki.soroush.uk/topic/instrumentation-plan/>)
