metricData, experimentation & growth

Sample Ratio Mismatch

Sample ratio mismatch is a statistically unlikely difference between an experiment’s intended allocation and its observed assignment or exposure counts.

Download Markdown (.md)

01What it means

Sample ratio mismatch, commonly abbreviated SRM, occurs when the observed number of experimental units in each variant deviates more from the planned allocation than random variation reasonably explains. A 50/50 experiment might produce 50.1/49.9 by chance; a sufficiently improbable imbalance is a diagnostic signal that assignment, eligibility, exposure, logging, or data processing may be broken.[1][2]

Allocation integrity diagnostic Compare observed variant counts with the planned assignment ratio
Observed allocation ↔ planned allocation
Plan
Allocation proportions and randomization unit
Count
Eligible assigned or exposed units
Test
Goodness-of-fit with a strict alert threshold
Respond
Pause interpretation; explain and repair mismatch

A sufficiently improbable mismatch is a stop-and-investigate signal for assignment, eligibility, exposure, telemetry, joins, bots, failures, or exclusions.

02Why it matters

Randomization supports causal inference only when treatment and control are created and measured as intended. SRM can reveal bot filtering, dropped events, inconsistent eligibility, redirect failures, variant-specific crashes, or incorrect joins. An apparent treatment effect calculated after such selection can point in either direction, so checking outcomes before resolving SRM creates false confidence.

03When to use it

Run an SRM check for every randomized online experiment before interpreting treatment effects, using the planned allocation and the actual randomization unit. Also monitor ratios by important platform or eligibility boundaries when justified. The test is a warning, not a root-cause detector, and repeated monitoring requires an alerting design that accounts for repeated looks.

04How it works

  1. Record the intended allocation proportions and randomization unit before launch.
  2. Count eligible assigned or exposed units using data independent of outcome metrics.
  3. Compare observed counts with expected counts using an appropriate goodness-of-fit test and a deliberately strict alert threshold.
  4. If the check alerts, pause interpretation and investigate assignment, eligibility, exposure, telemetry, joins, bots, failures, and exclusions.
  5. Resume analysis only after explaining the imbalance, repairing the pipeline when possible, and documenting whether trustworthy inference remains feasible.

05A practical example

Example

A checkout experiment intends to split customer accounts equally, but analysis contains 47 percent control and 53 percent treatment. The team finds that an error in the control experience prevented some mobile events from reaching the exposure table. Because missingness depends on variant, it repairs the logging and reruns the experiment instead of adjusting the counts after the fact.

06Where it breaks

An SRM test can miss balanced failures that affect every variant or offsetting errors that preserve total counts. Very large experiments flag tiny discrepancies that may be harmless, while small samples have little power. Segment fishing after an alert can create noise, and deleting records merely to restore the expected ratio does not repair compromised randomization.

Common mistakes

  • Treating SRM as a minor metric anomaly while continuing to interpret experiment outcomes.
  • Testing the wrong unit, such as events when randomization occurred by account.
  • Forcing counts back to the planned ratio through exclusions without identifying why records differ.

07Sources

  1. Diagnosing Sample Ratio Mismatch in Online Controlled ExperimentsMicrosoft Research · Research paperDiagnosing Sample Ratio Mismatch in Online Controlled Experiments
  2. Diagnosing Sample Ratio Mismatch in A/B TestingMicrosoft Research · Research explanationDiagnosing Sample Ratio Mismatch in A/B Testing
View saved topics

Start typing to search titles, definitions, categories, and frameworks.