# Statistical Power

Statistical power is the probability that a specified test detects an effect of a specified size when that effect exists under its assumptions.

- Type: metric
- Domain: Data, experimentation & growth
- Level: Advanced
- Last reviewed: 2026-10-05
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/statistical-power/>)

## What it means

Power describes a study design’s sensitivity to an assumed alternative, commonly expressed as one minus the false-negative rate. It depends on sample size, variability, allocation, test threshold, analysis method, and the effect size chosen for planning; it is not a property of a completed result alone.

## Why it matters

An underpowered experiment can miss useful improvements and produce unstable effect estimates. Planning power forces teams to connect traffic and duration to the decision they need to make.

## When to use it

Use power before launching a test or study, after defining a practically meaningful effect and valid analysis plan. Revisit it only for documented design changes; do not calculate post hoc power as a substitute for reporting uncertainty.

## How it works

1. Specify the estimand, primary metric, error rate, and analysis method.
2. Choose a practically meaningful alternative effect and estimate baseline variability.
3. Calculate required sample size or achievable power for the planned allocation and duration.
4. Check whether traffic, seasonality, and exposure constraints make the plan feasible.
5. Report the design sensitivity with the final effect estimate and interval.

## A practical example

A team wants to detect a one-point reduction in failed verification, not any nonzero movement. Its power calculation shows the planned two-week test cannot do that reliably, so it extends the duration and narrows the eligible population definition.

## Where it breaks

Power inherits assumptions about baseline rates, variance, and behavior. Higher planned power cannot correct bias, bad instrumentation, or an irrelevant metric, and a null result remains uncertain rather than proof of no effect.

## Common mistakes

- Planning for an implausibly large effect to shorten a test.
- Calling a completed nonsignificant test proof of no difference.
- Using post hoc power instead of an interval and design review.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/statistical-power/#visualization-statistical-power>)

## Sources

1. [Engineering Statistics Handbook](<https://www.itl.nist.gov/div898/handbook/>) — National Institute of Standards and Technology · Government statistical reference
2. [ASA Statement on Statistical Significance and P-Values](<https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf>) — American Statistical Association · Professional society statement

## Related topics

- [Sample Size](<https://wiki.soroush.uk/topic/sample-size/>)
- [Minimum Detectable Effect](<https://wiki.soroush.uk/topic/minimum-detectable-effect/>)
- [Confidence Interval](<https://wiki.soroush.uk/topic/confidence-interval/>)
