# Statistical Significance

Statistical significance describes how incompatible observed data are with a specified statistical model, not whether a product result is important or true.

- Type: term
- Domain: Data, experimentation & growth
- Level: Intermediate
- Last reviewed: 2026-10-05
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/statistical-significance/>)

## What it means

A result is commonly called statistically significant when its p-value is below a preselected threshold under a stated null model. It is a property of a test, its assumptions, and its analysis plan—not a measure of effect size, practical value, replication probability, or the probability that a hypothesis is true.

## Why it matters

Product experiments often turn a threshold crossing into a ship decision. Understanding the boundary prevents teams from presenting noise as certainty or a tiny detectable change as a meaningful customer or business improvement.

## When to use it

Use the term when reporting a prespecified statistical test alongside an effect estimate, confidence interval, practical threshold, and data-quality checks. Do not use it as the only decision rule, especially for high-stakes, small-sample, or repeatedly monitored experiments.

## How it works

1. Define the decision, primary metric, comparison, and analysis plan before observing outcomes.
2. Specify a null model and an error threshold appropriate to the decision.
3. Collect data under the planned assignment and check instrumentation and allocation quality.
4. Calculate the test result with its assumptions, effect estimate, and interval.
5. Interpret the result with practical impact, uncertainty, guardrails, and replication context.

## A practical example

A search change increases completed searches by 0.03 percentage points with a low p-value. The team calls the result statistically detectable but does not ship immediately: the gain is below its practical threshold and the mobile interval still includes a meaningful loss.

## Where it breaks

The label depends on sample size, model assumptions, stopping behavior, multiplicity, and reporting choices. A nonsignificant result is not proof of no effect, and a significant result can be trivial, biased, or non-reproducible.

## Common mistakes

- Treating statistical significance as proof that a variant is better.
- Equating a threshold crossing with a large or valuable effect.
- Ignoring repeated looks, multiple metrics, or failed experiment-quality checks.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/statistical-significance/#visualization-statistical-significance>)

## Sources

1. [ASA Statement on Statistical Significance and P-Values](<https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf>) — American Statistical Association · Professional society statement
2. [Engineering Statistics Handbook](<https://www.itl.nist.gov/div898/handbook/>) — National Institute of Standards and Technology · Government statistical reference

## Related topics

- [P-Value](<https://wiki.soroush.uk/topic/p-value/>)
- [Practical Significance](<https://wiki.soroush.uk/topic/practical-significance/>)
- [Confidence Interval](<https://wiki.soroush.uk/topic/confidence-interval/>)
