# P-Value

A p-value measures how incompatible observed data are with a specified statistical model, often including a null hypothesis.

- Type: metric
- Domain: Data, experimentation & growth
- Level: Intermediate
- Last reviewed: 2026-10-05
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/p-value/>)

## What it means

A p-value is the probability, calculated under a specified model and null hypothesis, of observing data at least as incompatible with that model as the data observed. It is not the probability that the null is true, the probability a result arose by chance alone, or the size and importance of an effect.

## Why it matters

P-values are common in experiment dashboards and easy to overread. Correct interpretation keeps a product team from shipping an illusory lift, rejecting useful uncertainty, or treating a threshold as a complete decision system.

## When to use it

Use a p-value only within a prespecified, valid analysis that states the model, hypotheses, stopping rule, and error control. Pair it with an effect estimate, interval, practical threshold, and diagnostics; use appropriate sequential or multiple-testing methods when applicable.

## How it works

1. Specify the null model, metric, test statistic, and analysis horizon.
2. Collect data using the planned assignment and eligibility rules.
3. Calculate how extreme the observed statistic is under the null model.
4. Report the p-value with effect size, interval, assumptions, and decision context.
5. Investigate data-quality or design failures before interpreting a small value.

## A practical example

An onboarding variant has p=0.03 for its primary completion metric but the estimated lift is 0.1 points and its interval includes zero. The team reports compatible evidence against the exact no-effect model, not proof that the experience creates meaningful activation.

## Where it breaks

Values depend on sample size, modeling choices, stopping, and the set of analyses attempted. They cannot quantify product value or the probability a hypothesis is correct, and they do not repair selection bias or failed randomization.

## Common mistakes

- Saying the p-value is the probability that the null hypothesis is true.
- Treating p\<0.05 as a universal shipping rule.
- Hiding metrics, looks, or segments tried before selecting a result.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/p-value/#visualization-p-value>)

## Sources

1. [ASA Statement on Statistical Significance and P-Values](<https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf>) — American Statistical Association · Professional society statement
2. [Engineering Statistics Handbook](<https://www.itl.nist.gov/div898/handbook/>) — National Institute of Standards and Technology · Government statistical reference

## Related topics

- [Statistical Significance](<https://wiki.soroush.uk/topic/statistical-significance/>)
- [Confidence Interval](<https://wiki.soroush.uk/topic/confidence-interval/>)
- [Sequential Testing](<https://wiki.soroush.uk/topic/sequential-testing/>)
