# Null and Alternative Hypotheses

Null and alternative hypotheses specify the statistical comparison a test evaluates and the effect pattern it is designed to detect.

- Type: framework
- Domain: Data, experimentation & growth
- Level: Intermediate
- Last reviewed: 2026-10-05
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/null-and-alternative-hypotheses/>)

## What it means

A null hypothesis states a reference model, often no difference between variants, while an alternative specifies the departure of interest. The pair structures a statistical test; it does not replace a product hypothesis about users, mechanisms, value, or implementation risk.

## Why it matters

Clear hypotheses prevent metric fishing and make p-values, power, and stopping rules interpretable. They also reveal when a product question is too vague to be answered by a single test.

## When to use it

Define a null and alternative before formal hypothesis testing. Pair them with a product hypothesis, primary metric, population, directionality, and practical threshold; avoid adding alternatives after observing the data.

## How it works

1. State the product decision and measurable estimand.
2. Write the reference null and the alternative effect pattern.
3. Choose a one- or two-sided test only with prior justification.
4. Set error rates, power, and analysis rules.
5. Report the test alongside effect size, uncertainty, and decision context.

## A practical example

Before testing a new reminder, a team states H0 as no change in weekly completion and H1 as an increase. It also declares that only an increase of at least one point would justify notification cost, separating the statistical model from the shipping rule.

## Where it breaks

A simple null can be unrealistic, and rejecting it does not identify the mechanism or prove broad product value. Hypothesis tests can be misleading with poor measurement, selection, or unplanned multiplicity.

## Common mistakes

- Writing hypotheses after seeing outcomes.
- Treating failure to reject the null as proof it is true.
- Confusing a product hypothesis with a statistical alternative.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/null-and-alternative-hypotheses/#visualization-null-and-alternative-hypotheses>)

## Sources

1. [Engineering Statistics Handbook](<https://www.itl.nist.gov/div898/handbook/>) — National Institute of Standards and Technology · Government statistical reference
2. [ASA Statement on Statistical Significance and P-Values](<https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf>) — American Statistical Association · Professional society statement

## Related topics

- [P-Value](<https://wiki.soroush.uk/topic/p-value/>)
- [Statistical Significance](<https://wiki.soroush.uk/topic/statistical-significance/>)
- [Product Hypothesis](<https://wiki.soroush.uk/topic/product-hypothesis/>)
