Statistical Significance
Statistical significance describes how incompatible observed data are with a specified statistical model, not whether a product result is important or true.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/statistical-significance/
01What it means
A result is commonly called statistically significant when its p-value is below a preselected threshold under a stated null model. It is a property of a test, its assumptions, and its analysis plan—not a measure of effect size, practical value, replication probability, or the probability that a hypothesis is true.[1][2]
- 01Predefine analysisDecision, metric, comparison, horizon
- 02Set null + thresholdModel and error tolerance
- 03Collect + verifyAssignment, instrumentation, allocation
- 04Calculate testEffect estimate, interval, assumptions
- 05Interpret decisionPractical impact, guardrails, uncertainty
Statistical significance is not effect size, practical value, replication probability, or the probability that a hypothesis is true.
02Why it matters
Product experiments often turn a threshold crossing into a ship decision. Understanding the boundary prevents teams from presenting noise as certainty or a tiny detectable change as a meaningful customer or business improvement.
03When to use it
Use the term when reporting a prespecified statistical test alongside an effect estimate, confidence interval, practical threshold, and data-quality checks. Do not use it as the only decision rule, especially for high-stakes, small-sample, or repeatedly monitored experiments.
04How it works
- Define the decision, primary metric, comparison, and analysis plan before observing outcomes.
- Specify a null model and an error threshold appropriate to the decision.
- Collect data under the planned assignment and check instrumentation and allocation quality.
- Calculate the test result with its assumptions, effect estimate, and interval.
- Interpret the result with practical impact, uncertainty, guardrails, and replication context.
05A practical example
A search change increases completed searches by 0.03 percentage points with a low p-value. The team calls the result statistically detectable but does not ship immediately: the gain is below its practical threshold and the mobile interval still includes a meaningful loss.
06Where it breaks
The label depends on sample size, model assumptions, stopping behavior, multiplicity, and reporting choices. A nonsignificant result is not proof of no effect, and a significant result can be trivial, biased, or non-reproducible.
Common mistakes
- Treating statistical significance as proof that a variant is better.
- Equating a threshold crossing with a large or valuable effect.
- Ignoring repeated looks, multiple metrics, or failed experiment-quality checks.
07Sources
- ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values
- Engineering Statistics HandbookNational Institute of Standards and Technology · Government statistical referenceEngineering Statistics Handbook