methodLeadership & teams

A/B Testing

A/B testing randomly assigns eligible units to variants and compares predefined outcomes to estimate the causal effect of a product change.

Download Markdown (.md)

01What it means

A/B testing is a randomized controlled experiment in which eligible users, sessions, accounts, or other units are assigned to a control experience or one or more variants. When assignment, exposure, measurement, and analysis are sound, differences in outcomes estimate the causal impact of the tested change for the studied population and period.[1][2][3]

Controlled experiment Random assignment makes the comparison causal
  1. 01HypothesisMetric and guardrails
  2. 02RandomizeControl and variant
  3. 03ExposeConsistent treatment
  4. 04CompareEffect and uncertainty
  5. 05DecideDocument and monitor

Credibility depends on preserving assignment, exposure, measurement, and analysis.

02Why it matters

Behavioral metrics are influenced by seasonality, selection, marketing, novelty, and many other forces. Randomization helps separate a change’s effect from those confounders. A/B tests can reduce reliance on executive opinion, reveal unintended tradeoffs, and show that an apparently obvious design either has little effect or harms a guardrail metric.

03When to use it

Use A/B testing for measurable, ethically acceptable, reversible changes when there is enough traffic, stable instrumentation, and genuine uncertainty. It is not a substitute for discovery or usability research, and it is often inappropriate for rare harms, tiny populations, network effects, legally required experiences, or changes users cannot meaningfully avoid.

04How it works

  1. Write a falsifiable hypothesis, define the eligible population and randomization unit, and preselect a primary metric plus guardrails.
  2. Estimate detectable effect, sample size, duration, and stopping rules before launch; verify logging and run an A/A or quality check when warranted.
  3. Assign units consistently, isolate variants, monitor exposure, sample-ratio mismatch, data quality, safety, and operational failures without peeking for convenient significance.
  4. Analyze effect sizes and uncertainty, segment only with justified plans, and examine guardrails, novelty, heterogeneity, and practical—not merely statistical—importance.
  5. Document the decision and limitations, roll out cautiously, and continue monitoring because an experiment result does not guarantee durable or universal impact.

05A practical example

Example

A learning platform tests a shorter signup form. Completion rises, but the team’s predefined guardrails show more underage accounts and more support contacts about consent. It does not ship the variant. Instead, it keeps necessary age and guardian steps, rewrites confusing questions, and follows with usability research before another test.

06Where it breaks

Experiments answer a narrow causal question under particular conditions. They can miss long-term effects, spillovers, small but severe harms, and impacts on people absent from the sample. Multiple testing, peeking, broken randomization, selective reporting, and proxy metrics produce false confidence. Ethical acceptability must be decided before statistical convenience.

Common mistakes

  • Choosing success metrics after seeing which numbers moved.
  • Stopping as soon as a result crosses a significance threshold or ignoring sample-ratio mismatch.
  • Optimizing conversion while omitting guardrails for retention, accessibility, complaints, fairness, or customer harm.

07Sources

  1. Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at ScaleMicrosoft Research · Research paperThree Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale
  2. Online Experimentation at MicrosoftMicrosoft Research · Research paperOnline Experimentation at Microsoft
  3. A/B Testing Best Practices for SearchGoogle for Developers · Technical guidanceA/B Testing Best Practices for Search
View saved topics

Start typing to search titles, definitions, categories, and frameworks.