A/A Testing
A/A testing compares nominally identical experiment variants to check assignment, exposure, metrics, and analysis behavior before relying on a new experiment system.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/a-a-testing/
01What it means
An A/A test assigns eligible units to two conditions that should receive the same experience. It tests whether randomization, logging, metric computation, and statistical procedures behave as expected under no intended treatment difference.[1][2]
- 01Eligible populationSame inclusion rules
- 02Random assignmentTwo nominal conditions
- 03Identical experienceNo intended treatment difference
- 04Full analysis pipelineLogging, metrics, statistical procedure
- 05Diagnose anomaliesRepair before interpreting experiments
Unexpected allocation, baseline, missingness, or false-positive behavior must be explained before trusting later A/B results.
02Why it matters
Experiment platforms can produce misleading results through assignment faults, event loss, and analysis errors. A/A tests make these failures observable before they distort product decisions.
03When to use it
Use A/A tests when launching or materially changing experimentation infrastructure, assignment, telemetry, or analysis methods. Do not use them as a substitute for ongoing quality checks.
04How it works
- Define identical experiences and assignment.
- Run the complete eligibility and analysis pipeline.
- Check allocation, baseline metrics, missingness, and consistency.
- Evaluate false-positive behavior against the documented method.
- Repair anomalies before interpreting A/B results.
05A practical example
After an exposure-service migration, an A/A test finds a persistent mobile difference. The team discovers exposure is logged after a rendering failure and fixes the pipeline before a product test.
06Where it breaks
A passing A/A test cannot prove future experiments valid or expose all variant-specific defects. Small samples can miss problems, while many unplanned slices can create false alarms.
Common mistakes
- Treating an A/A test as feature evidence.
- Ignoring a detected mismatch.
- Searching many segments until one appears significant.
07Sources
- Diagnosing Sample Ratio Mismatch in Online Controlled ExperimentsMicrosoft Research · Research paperDiagnosing Sample Ratio Mismatch in Online Controlled Experiments
- NIST/SEMATECH e-Handbook of Statistical MethodsNational Institute of Standards and Technology · Government statistical referenceNIST/SEMATECH e-Handbook of Statistical Methods