Statistical Power
Statistical power is the probability that a specified test detects an effect of a specified size when that effect exists under its assumptions.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/statistical-power/
01What it means
Power describes a study design’s sensitivity to an assumed alternative, commonly expressed as one minus the false-negative rate. It depends on sample size, variability, allocation, test threshold, analysis method, and the effect size chosen for planning; it is not a property of a completed result alone.[1][2]
- Effect size
- Assumed decision-relevant alternative
- Sample size
- Independent eligible units
- Variability
- Baseline noise or event rate
- Allocation
- Distribution across conditions
- Test threshold
- False-positive tolerance
- Analysis method
- Estimator and design assumptions
Power belongs to a specified design and alternative; it is not a retrospective quality score for one completed result.
02Why it matters
An underpowered experiment can miss useful improvements and produce unstable effect estimates. Planning power forces teams to connect traffic and duration to the decision they need to make.
03When to use it
Use power before launching a test or study, after defining a practically meaningful effect and valid analysis plan. Revisit it only for documented design changes; do not calculate post hoc power as a substitute for reporting uncertainty.
04How it works
- Specify the estimand, primary metric, error rate, and analysis method.
- Choose a practically meaningful alternative effect and estimate baseline variability.
- Calculate required sample size or achievable power for the planned allocation and duration.
- Check whether traffic, seasonality, and exposure constraints make the plan feasible.
- Report the design sensitivity with the final effect estimate and interval.
05A practical example
A team wants to detect a one-point reduction in failed verification, not any nonzero movement. Its power calculation shows the planned two-week test cannot do that reliably, so it extends the duration and narrows the eligible population definition.
06Where it breaks
Power inherits assumptions about baseline rates, variance, and behavior. Higher planned power cannot correct bias, bad instrumentation, or an irrelevant metric, and a null result remains uncertain rather than proof of no effect.
Common mistakes
- Planning for an implausibly large effect to shorten a test.
- Calling a completed nonsignificant test proof of no difference.
- Using post hoc power instead of an interval and design review.
07Sources
- Engineering Statistics HandbookNational Institute of Standards and Technology · Government statistical referenceEngineering Statistics Handbook
- ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values