P-Value
A p-value measures how incompatible observed data are with a specified statistical model, often including a null hypothesis.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/p-value/
01What it means
A p-value is the probability, calculated under a specified model and null hypothesis, of observing data at least as incompatible with that model as the data observed. It is not the probability that the null is true, the probability a result arose by chance alone, or the size and importance of an effect.[1][2]
- 01Specify null modelMetric, statistic, assumptions, horizon
- 02Observe dataPlanned assignment and eligibility
- 03Measure extremenessData at least as incompatible under the null
- 04Calculate p-valueConditional on the model and procedure
- 05Interpret with contextEffect, interval, quality, practical decision
A p-value is not the probability the null is true, the probability of chance alone, or the size and importance of the effect.
02Why it matters
P-values are common in experiment dashboards and easy to overread. Correct interpretation keeps a product team from shipping an illusory lift, rejecting useful uncertainty, or treating a threshold as a complete decision system.
03When to use it
Use a p-value only within a prespecified, valid analysis that states the model, hypotheses, stopping rule, and error control. Pair it with an effect estimate, interval, practical threshold, and diagnostics; use appropriate sequential or multiple-testing methods when applicable.
04How it works
- Specify the null model, metric, test statistic, and analysis horizon.
- Collect data using the planned assignment and eligibility rules.
- Calculate how extreme the observed statistic is under the null model.
- Report the p-value with effect size, interval, assumptions, and decision context.
- Investigate data-quality or design failures before interpreting a small value.
05A practical example
An onboarding variant has p=0.03 for its primary completion metric but the estimated lift is 0.1 points and its interval includes zero. The team reports compatible evidence against the exact no-effect model, not proof that the experience creates meaningful activation.
06Where it breaks
Values depend on sample size, modeling choices, stopping, and the set of analyses attempted. They cannot quantify product value or the probability a hypothesis is correct, and they do not repair selection bias or failed randomization.
Common mistakes
- Saying the p-value is the probability that the null hypothesis is true.
- Treating p<0.05 as a universal shipping rule.
- Hiding metrics, looks, or segments tried before selecting a result.
07Sources
- ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values
- Engineering Statistics HandbookNational Institute of Standards and Technology · Government statistical referenceEngineering Statistics Handbook