Minimum Detectable Effect
A minimum detectable effect is the smallest effect an experiment is designed to detect with specified power, error control, and assumptions.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/minimum-detectable-effect/
01What it means
The minimum detectable effect, or MDE, is a design sensitivity target derived from sample size, variability, allocation, testing method, and error rates. It is not the smallest effect that could matter to the product, although the two should be compared.[1][2]
- Sample size
- Eligible independent units
- Variability
- Baseline noise or event rate
- Allocation
- How units split across conditions
- Error rates
- False-positive and false-negative targets
- Analysis method
- Test and estimator assumptions
- Practical threshold
- Effect large enough to change action
Compare MDE with the smallest practically meaningful effect; they answer different questions.
02Why it matters
MDE makes an experiment’s resolution visible before launch and prevents teams from treating a short test as able to distinguish a valuable small gain from no meaningful change.
03When to use it
Use MDE in experiment planning to determine whether available traffic can answer the product question. Redesign, lengthen, or decline the test when sensitivity is too coarse.
04How it works
- Define the primary metric and decision-relevant effect.
- Set power, error rates, allocation, and analysis horizon.
- Estimate baseline variability and eligible sample size.
- Calculate the MDE and compare it with the practical threshold.
- Document any justified design change before analysis.
05A practical example
A team can detect only a three-point conversion change in two weeks, while a one-point change would matter commercially. It extends the test rather than calling the shorter one inconclusive.
06Where it breaks
MDE depends on model assumptions and says neither whether an observed effect is real nor whether it is valuable. Optimistic variance or effect assumptions can make a plan misleading.
Common mistakes
- Confusing the MDE with the expected effect.
- Choosing an MDE larger than the decision-relevant effect.
- Ignoring traffic or variance changes after planning.
07Sources
- Engineering Statistics HandbookNational Institute of Standards and Technology · Government statistical referenceEngineering Statistics Handbook
- ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values