Experimentation
Product experimentation is a disciplined system for reducing uncertainty through hypotheses, appropriate tests, reliable measurement, and learning.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/experimentation/
01What it means
Experimentation is the organizational practice of turning important uncertainties into explicit hypotheses and selecting methods that can produce decision-relevant evidence. It includes—but is not limited to—A/B tests. Prototypes, usability studies, fake-door tests, simulations, pilots, and randomized trials answer different questions and require different safeguards.[1][2][3]
- 01Frame decisionMap assumptions, consequence, evidence gap
- 02Choose methodMatch observation to uncertainty
- 03Predefine safeguardsMeasures, guardrails, stopping, privacy, ownership
- 04Run + check qualityDocument deviations and limitations
- 05Update decisionRecord learning; continue, change, or stop
Experimentation includes prototypes, usability studies, simulations, pilots, fake doors, and randomized trials—not A/B testing alone.
02Why it matters
Teams can ship quickly while learning slowly if activity is mistaken for evidence. A healthy experimentation practice clarifies assumptions, matches methods to risk, preserves negative results, and improves decisions across discovery and delivery. It also creates a counterweight to hierarchy, provided leaders are willing to change direction when evidence disagrees.
03When to use it
Use experimentation when a consequential decision contains uncertainty that can be reduced with an ethical, feasible test. Choose the lightest method capable of answering the question. Do not experiment on rights, required protections, or deceptive conditions merely because the platform makes exposure easy.
04How it works
- Frame the decision and map assumptions about desirability, usability, feasibility, viability, safety, accessibility, and fairness.
- Prioritize uncertainties by consequence and evidence gap, then choose a method whose observations can genuinely update the decision.
- Define hypothesis, participants, measures, guardrails, stopping conditions, ownership, privacy controls, and failure responses before running the test.
- Execute with quality checks and document deviations; distinguish exploratory learning from confirmatory evidence.
- Synthesize what changed in the team’s belief, record null and negative findings, make the decision, and monitor what the experiment could not reveal.
05A practical example
A delivery service considers scheduled medication drop-offs. Before a citywide launch, it interviews patients and pharmacists, prototypes accessible scheduling, runs a limited operational pilot, and tests reminder timing. Safety incidents and missed handoffs are guardrails. Evidence leads to a pharmacy-confirmation step and human support rather than an automated rollout.
06Where it breaks
Experiments can become theater, metric chasing, or a way to offload risk onto users. Fast tests often measure easy proxies instead of durable value, and local results may not transfer across populations or contexts. A mature program needs statistical, qualitative, ethical, domain, privacy, and accessibility expertise—not only an experimentation platform.
Common mistakes
- Using an A/B test for every question, including those better answered by observation or usability research.
- Celebrating the number of experiments rather than the quality of learning and decisions.
- Discarding null results, undocumented failures, or evidence that conflicts with roadmap commitments.
07Sources
- The Anatomy of a Large-Scale Experimentation PlatformMicrosoft Research · Research paperThe Anatomy of a Large-Scale Experimentation Platform
- Online Experimentation at MicrosoftMicrosoft Research · Research paperOnline Experimentation at Microsoft
- Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at ScaleMicrosoft Research · Research paperThree Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale