# Confidence Interval

A confidence interval is a range produced by a specified repeated-sampling procedure that expresses uncertainty around an estimated quantity.

- Type: metric
- Domain: Data, experimentation & growth
- Level: Intermediate
- Last reviewed: 2026-10-05
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/confidence-interval/>)

## What it means

A confidence interval gives a lower and upper bound around an estimate, such as a conversion-rate difference, using a stated method and confidence level. Under repeated use of a valid 95% procedure, about 95% of such intervals would contain the fixed parameter; it is not a 95% probability statement about this one interval without a Bayesian model.

## Why it matters

Intervals show direction, magnitude, precision, and decision-relevant possibilities that a pass/fail result hides. They help teams see whether a test rules out a worthwhile improvement or a harmful regression.

## When to use it

Report intervals for estimates used in product decisions, especially experiment effects, rates, and forecasts. Ensure the method matches the metric, randomization unit, dependence, and monitoring plan; do not use a default interval to excuse a weak design.

## How it works

1. Define the population, estimand, and comparison.
2. Choose an interval method suited to the data and analysis plan.
3. Estimate the effect and sampling uncertainty.
4. Calculate and display the interval with its confidence level and assumptions.
5. Compare the plausible range with practical benefit and harm thresholds.

## A practical example

A payment-flow test estimates a 0.6-point conversion lift with a 95% interval from -0.2 to 1.4 points. Rather than declare success from the positive point estimate, the team recognizes that both a small loss and a useful lift remain plausible and continues to the planned sample.

## Where it breaks

Coverage relies on the design and model being appropriate. Intervals do not address bias, bad logging, selective analysis, or interference, and their meaning changes when teams repeatedly inspect data without a valid sequential procedure.

## Common mistakes

- Reading a 95% interval as a 95% probability about a fixed parameter.
- Reporting intervals without the estimate or confidence level.
- Ignoring an interval that includes decision-relevant harm.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/confidence-interval/#visualization-confidence-interval>)

## Sources

1. [What are confidence intervals?](<https://www.itl.nist.gov/div898/handbook/prc/section1/prc14.htm>) — National Institute of Standards and Technology · Government statistical reference
2. [ASA Statement on Statistical Significance and P-Values](<https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf>) — American Statistical Association · Professional society statement

## Related topics

- [P-Value](<https://wiki.soroush.uk/topic/p-value/>)
- [Statistical Power](<https://wiki.soroush.uk/topic/statistical-power/>)
- [Minimum Detectable Effect](<https://wiki.soroush.uk/topic/minimum-detectable-effect/>)
