Sample Ratio Mismatch
Sample ratio mismatch is a statistically unlikely difference between an experiment’s intended allocation and its observed assignment or exposure counts.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/sample-ratio-mismatch/
01What it means
Sample ratio mismatch, commonly abbreviated SRM, occurs when the observed number of experimental units in each variant deviates more from the planned allocation than random variation reasonably explains. A 50/50 experiment might produce 50.1/49.9 by chance; a sufficiently improbable imbalance is a diagnostic signal that assignment, eligibility, exposure, logging, or data processing may be broken.[1][2]
- Plan
- Allocation proportions and randomization unit
- Count
- Eligible assigned or exposed units
- Test
- Goodness-of-fit with a strict alert threshold
- Respond
- Pause interpretation; explain and repair mismatch
A sufficiently improbable mismatch is a stop-and-investigate signal for assignment, eligibility, exposure, telemetry, joins, bots, failures, or exclusions.
02Why it matters
Randomization supports causal inference only when treatment and control are created and measured as intended. SRM can reveal bot filtering, dropped events, inconsistent eligibility, redirect failures, variant-specific crashes, or incorrect joins. An apparent treatment effect calculated after such selection can point in either direction, so checking outcomes before resolving SRM creates false confidence.
03When to use it
Run an SRM check for every randomized online experiment before interpreting treatment effects, using the planned allocation and the actual randomization unit. Also monitor ratios by important platform or eligibility boundaries when justified. The test is a warning, not a root-cause detector, and repeated monitoring requires an alerting design that accounts for repeated looks.
04How it works
- Record the intended allocation proportions and randomization unit before launch.
- Count eligible assigned or exposed units using data independent of outcome metrics.
- Compare observed counts with expected counts using an appropriate goodness-of-fit test and a deliberately strict alert threshold.
- If the check alerts, pause interpretation and investigate assignment, eligibility, exposure, telemetry, joins, bots, failures, and exclusions.
- Resume analysis only after explaining the imbalance, repairing the pipeline when possible, and documenting whether trustworthy inference remains feasible.
05A practical example
A checkout experiment intends to split customer accounts equally, but analysis contains 47 percent control and 53 percent treatment. The team finds that an error in the control experience prevented some mobile events from reaching the exposure table. Because missingness depends on variant, it repairs the logging and reruns the experiment instead of adjusting the counts after the fact.
06Where it breaks
An SRM test can miss balanced failures that affect every variant or offsetting errors that preserve total counts. Very large experiments flag tiny discrepancies that may be harmless, while small samples have little power. Segment fishing after an alert can create noise, and deleting records merely to restore the expected ratio does not repair compromised randomization.
Common mistakes
- Treating SRM as a minor metric anomaly while continuing to interpret experiment outcomes.
- Testing the wrong unit, such as events when randomization occurred by account.
- Forcing counts back to the planned ratio through exclusions without identifying why records differ.
07Sources
- Diagnosing Sample Ratio Mismatch in Online Controlled ExperimentsMicrosoft Research · Research paperDiagnosing Sample Ratio Mismatch in Online Controlled Experiments
- Diagnosing Sample Ratio Mismatch in A/B TestingMicrosoft Research · Research explanationDiagnosing Sample Ratio Mismatch in A/B Testing