Confidence Interval
A confidence interval is a range produced by a specified repeated-sampling procedure that expresses uncertainty around an estimated quantity.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/confidence-interval/
01What it means
A confidence interval gives a lower and upper bound around an estimate, such as a conversion-rate difference, using a stated method and confidence level. Under repeated use of a valid 95% procedure, about 95% of such intervals would contain the fixed parameter; it is not a 95% probability statement about this one interval without a Bayesian model.[1][2]
Conceptual interval only. A valid 95% procedure contains the fixed parameter in about 95% of repeated samples; this is not a probability statement about one frequentist interval.
02Why it matters
Intervals show direction, magnitude, precision, and decision-relevant possibilities that a pass/fail result hides. They help teams see whether a test rules out a worthwhile improvement or a harmful regression.
03When to use it
Report intervals for estimates used in product decisions, especially experiment effects, rates, and forecasts. Ensure the method matches the metric, randomization unit, dependence, and monitoring plan; do not use a default interval to excuse a weak design.
04How it works
- Define the population, estimand, and comparison.
- Choose an interval method suited to the data and analysis plan.
- Estimate the effect and sampling uncertainty.
- Calculate and display the interval with its confidence level and assumptions.
- Compare the plausible range with practical benefit and harm thresholds.
05A practical example
A payment-flow test estimates a 0.6-point conversion lift with a 95% interval from -0.2 to 1.4 points. Rather than declare success from the positive point estimate, the team recognizes that both a small loss and a useful lift remain plausible and continues to the planned sample.
06Where it breaks
Coverage relies on the design and model being appropriate. Intervals do not address bias, bad logging, selective analysis, or interference, and their meaning changes when teams repeatedly inspect data without a valid sequential procedure.
Common mistakes
- Reading a 95% interval as a 95% probability about a fixed parameter.
- Reporting intervals without the estimate or confidence level.
- Ignoring an interval that includes decision-relevant harm.
07Sources
- What are confidence intervals?National Institute of Standards and Technology · Government statistical referenceWhat are confidence intervals?
- ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values