# Experimentation

Product experimentation is a disciplined system for reducing uncertainty through hypotheses, appropriate tests, reliable measurement, and learning.

- Type: practice
- Domain: Leadership & teams
- Level: Advanced
- Last reviewed: 2026-09-21
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/experimentation/>)

## What it means

Experimentation is the organizational practice of turning important uncertainties into explicit hypotheses and selecting methods that can produce decision-relevant evidence. It includes—but is not limited to—A/B tests. Prototypes, usability studies, fake-door tests, simulations, pilots, and randomized trials answer different questions and require different safeguards.

## Why it matters

Teams can ship quickly while learning slowly if activity is mistaken for evidence. A healthy experimentation practice clarifies assumptions, matches methods to risk, preserves negative results, and improves decisions across discovery and delivery. It also creates a counterweight to hierarchy, provided leaders are willing to change direction when evidence disagrees.

## When to use it

Use experimentation when a consequential decision contains uncertainty that can be reduced with an ethical, feasible test. Choose the lightest method capable of answering the question. Do not experiment on rights, required protections, or deceptive conditions merely because the platform makes exposure easy.

## How it works

1. Frame the decision and map assumptions about desirability, usability, feasibility, viability, safety, accessibility, and fairness.
2. Prioritize uncertainties by consequence and evidence gap, then choose a method whose observations can genuinely update the decision.
3. Define hypothesis, participants, measures, guardrails, stopping conditions, ownership, privacy controls, and failure responses before running the test.
4. Execute with quality checks and document deviations; distinguish exploratory learning from confirmatory evidence.
5. Synthesize what changed in the team’s belief, record null and negative findings, make the decision, and monitor what the experiment could not reveal.

## A practical example

A delivery service considers scheduled medication drop-offs. Before a citywide launch, it interviews patients and pharmacists, prototypes accessible scheduling, runs a limited operational pilot, and tests reminder timing. Safety incidents and missed handoffs are guardrails. Evidence leads to a pharmacy-confirmation step and human support rather than an automated rollout.

## Where it breaks

Experiments can become theater, metric chasing, or a way to offload risk onto users. Fast tests often measure easy proxies instead of durable value, and local results may not transfer across populations or contexts. A mature program needs statistical, qualitative, ethical, domain, privacy, and accessibility expertise—not only an experimentation platform.

## Common mistakes

- Using an A/B test for every question, including those better answered by observation or usability research.
- Celebrating the number of experiments rather than the quality of learning and decisions.
- Discarding null results, undocumented failures, or evidence that conflicts with roadmap commitments.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/experimentation/#visualization-experimentation>)

## Sources

1. [The Anatomy of a Large-Scale Experimentation Platform](<https://www.microsoft.com/en-us/research/publication/the-anatomy-of-a-large-scale-experimentation-platform/>) — Microsoft Research · Research paper
2. [Online Experimentation at Microsoft](<https://www.microsoft.com/en-us/research/publication/online-experimentation-at-microsoft/>) — Microsoft Research · Research paper
3. [Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale](<https://www.microsoft.com/en-us/research/publication/three-key-checklists-and-remedies-for-trustworthy-analysis-of-online-controlled-experiments-at-scale/>) — Microsoft Research · Research paper

## Related topics

- [A/B Testing](<https://wiki.soroush.uk/topic/a-b-testing/>)
- [Product Discovery](<https://wiki.soroush.uk/topic/product-discovery/>)
- [HEART Framework](<https://wiki.soroush.uk/topic/heart-framework/>)
