metricData, experimentation & growth

Confidence Interval

A confidence interval is a range produced by a specified repeated-sampling procedure that expresses uncertainty around an estimated quantity.

Download Markdown (.md)

01What it means

A confidence interval gives a lower and upper bound around an estimate, such as a conversion-rate difference, using a stated method and confidence level. Under repeated use of a valid 95% procedure, about 95% of such intervals would contain the fixed parameter; it is not a 95% probability statement about this one interval without a Bayesian model.[1][2]

Estimate plus sampling uncertainty Read the plausible effect range against practical benefit and harm thresholds

Conceptual interval only. A valid 95% procedure contains the fixed parameter in about 95% of repeated samples; this is not a probability statement about one frequentist interval.

02Why it matters

Intervals show direction, magnitude, precision, and decision-relevant possibilities that a pass/fail result hides. They help teams see whether a test rules out a worthwhile improvement or a harmful regression.

03When to use it

Report intervals for estimates used in product decisions, especially experiment effects, rates, and forecasts. Ensure the method matches the metric, randomization unit, dependence, and monitoring plan; do not use a default interval to excuse a weak design.

04How it works

  1. Define the population, estimand, and comparison.
  2. Choose an interval method suited to the data and analysis plan.
  3. Estimate the effect and sampling uncertainty.
  4. Calculate and display the interval with its confidence level and assumptions.
  5. Compare the plausible range with practical benefit and harm thresholds.

05A practical example

Example

A payment-flow test estimates a 0.6-point conversion lift with a 95% interval from -0.2 to 1.4 points. Rather than declare success from the positive point estimate, the team recognizes that both a small loss and a useful lift remain plausible and continues to the planned sample.

06Where it breaks

Coverage relies on the design and model being appropriate. Intervals do not address bias, bad logging, selective analysis, or interference, and their meaning changes when teams repeatedly inspect data without a valid sequential procedure.

Common mistakes

  • Reading a 95% interval as a 95% probability about a fixed parameter.
  • Reporting intervals without the estimate or confidence level.
  • Ignoring an interval that includes decision-relevant harm.

07Sources

  1. What are confidence intervals?National Institute of Standards and Technology · Government statistical referenceWhat are confidence intervals?
  2. ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values
View saved topics

Start typing to search titles, definitions, categories, and frameworks.