termData, experimentation & growth

Statistical Significance

Statistical significance describes how incompatible observed data are with a specified statistical model, not whether a product result is important or true.

Download Markdown (.md)

01What it means

A result is commonly called statistically significant when its p-value is below a preselected threshold under a stated null model. It is a property of a test, its assumptions, and its analysis plan—not a measure of effect size, practical value, replication probability, or the probability that a hypothesis is true.[1][2]

A result of a specified test Plan the null and threshold, verify the data, then interpret beyond the label
  1. 01Predefine analysisDecision, metric, comparison, horizon
  2. 02Set null + thresholdModel and error tolerance
  3. 03Collect + verifyAssignment, instrumentation, allocation
  4. 04Calculate testEffect estimate, interval, assumptions
  5. 05Interpret decisionPractical impact, guardrails, uncertainty

Statistical significance is not effect size, practical value, replication probability, or the probability that a hypothesis is true.

02Why it matters

Product experiments often turn a threshold crossing into a ship decision. Understanding the boundary prevents teams from presenting noise as certainty or a tiny detectable change as a meaningful customer or business improvement.

03When to use it

Use the term when reporting a prespecified statistical test alongside an effect estimate, confidence interval, practical threshold, and data-quality checks. Do not use it as the only decision rule, especially for high-stakes, small-sample, or repeatedly monitored experiments.

04How it works

  1. Define the decision, primary metric, comparison, and analysis plan before observing outcomes.
  2. Specify a null model and an error threshold appropriate to the decision.
  3. Collect data under the planned assignment and check instrumentation and allocation quality.
  4. Calculate the test result with its assumptions, effect estimate, and interval.
  5. Interpret the result with practical impact, uncertainty, guardrails, and replication context.

05A practical example

Example

A search change increases completed searches by 0.03 percentage points with a low p-value. The team calls the result statistically detectable but does not ship immediately: the gain is below its practical threshold and the mobile interval still includes a meaningful loss.

06Where it breaks

The label depends on sample size, model assumptions, stopping behavior, multiplicity, and reporting choices. A nonsignificant result is not proof of no effect, and a significant result can be trivial, biased, or non-reproducible.

Common mistakes

  • Treating statistical significance as proof that a variant is better.
  • Equating a threshold crossing with a large or valuable effect.
  • Ignoring repeated looks, multiple metrics, or failed experiment-quality checks.

07Sources

  1. ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values
  2. Engineering Statistics HandbookNational Institute of Standards and Technology · Government statistical referenceEngineering Statistics Handbook
View saved topics

Start typing to search titles, definitions, categories, and frameworks.