metricData, experimentation & growth

Statistical Power

Statistical power is the probability that a specified test detects an effect of a specified size when that effect exists under its assumptions.

Download Markdown (.md)

01What it means

Power describes a study design’s sensitivity to an assumed alternative, commonly expressed as one minus the false-negative rate. It depends on sample size, variability, allocation, test threshold, analysis method, and the effect size chosen for planning; it is not a property of a completed result alone.[1][2]

Sensitivity to one assumed alternative Power changes with effect size, sample, noise, allocation, and the test plan
Statistical power = 1 − false-negative rate
Effect size
Assumed decision-relevant alternative
Sample size
Independent eligible units
Variability
Baseline noise or event rate
Allocation
Distribution across conditions
Test threshold
False-positive tolerance
Analysis method
Estimator and design assumptions

Power belongs to a specified design and alternative; it is not a retrospective quality score for one completed result.

02Why it matters

An underpowered experiment can miss useful improvements and produce unstable effect estimates. Planning power forces teams to connect traffic and duration to the decision they need to make.

03When to use it

Use power before launching a test or study, after defining a practically meaningful effect and valid analysis plan. Revisit it only for documented design changes; do not calculate post hoc power as a substitute for reporting uncertainty.

04How it works

  1. Specify the estimand, primary metric, error rate, and analysis method.
  2. Choose a practically meaningful alternative effect and estimate baseline variability.
  3. Calculate required sample size or achievable power for the planned allocation and duration.
  4. Check whether traffic, seasonality, and exposure constraints make the plan feasible.
  5. Report the design sensitivity with the final effect estimate and interval.

05A practical example

Example

A team wants to detect a one-point reduction in failed verification, not any nonzero movement. Its power calculation shows the planned two-week test cannot do that reliably, so it extends the duration and narrows the eligible population definition.

06Where it breaks

Power inherits assumptions about baseline rates, variance, and behavior. Higher planned power cannot correct bias, bad instrumentation, or an irrelevant metric, and a null result remains uncertain rather than proof of no effect.

Common mistakes

  • Planning for an implausibly large effect to shorten a test.
  • Calling a completed nonsignificant test proof of no difference.
  • Using post hoc power instead of an interval and design review.

07Sources

  1. Engineering Statistics HandbookNational Institute of Standards and Technology · Government statistical referenceEngineering Statistics Handbook
  2. ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values
View saved topics

Start typing to search titles, definitions, categories, and frameworks.