metricData, experimentation & growth

Minimum Detectable Effect

A minimum detectable effect is the smallest effect an experiment is designed to detect with specified power, error control, and assumptions.

Download Markdown (.md)

01What it means

The minimum detectable effect, or MDE, is a design sensitivity target derived from sample size, variability, allocation, testing method, and error rates. It is not the smallest effect that could matter to the product, although the two should be compared.[1][2]

Planned sensitivity, not product importance Design choices determine the smallest effect the study is powered to detect
Minimum detectable effect
Sample size
Eligible independent units
Variability
Baseline noise or event rate
Allocation
How units split across conditions
Error rates
False-positive and false-negative targets
Analysis method
Test and estimator assumptions
Practical threshold
Effect large enough to change action

Compare MDE with the smallest practically meaningful effect; they answer different questions.

02Why it matters

MDE makes an experiment’s resolution visible before launch and prevents teams from treating a short test as able to distinguish a valuable small gain from no meaningful change.

03When to use it

Use MDE in experiment planning to determine whether available traffic can answer the product question. Redesign, lengthen, or decline the test when sensitivity is too coarse.

04How it works

  1. Define the primary metric and decision-relevant effect.
  2. Set power, error rates, allocation, and analysis horizon.
  3. Estimate baseline variability and eligible sample size.
  4. Calculate the MDE and compare it with the practical threshold.
  5. Document any justified design change before analysis.

05A practical example

Example

A team can detect only a three-point conversion change in two weeks, while a one-point change would matter commercially. It extends the test rather than calling the shorter one inconclusive.

06Where it breaks

MDE depends on model assumptions and says neither whether an observed effect is real nor whether it is valuable. Optimistic variance or effect assumptions can make a plan misleading.

Common mistakes

  • Confusing the MDE with the expected effect.
  • Choosing an MDE larger than the decision-relevant effect.
  • Ignoring traffic or variance changes after planning.

07Sources

  1. Engineering Statistics HandbookNational Institute of Standards and Technology · Government statistical referenceEngineering Statistics Handbook
  2. ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values
View saved topics

Start typing to search titles, definitions, categories, and frameworks.