# Sample Size

Sample size is the number of independent analysis units included in a study or experiment, chosen to support a stated precision or detection goal.

- Type: metric
- Domain: Data, experimentation & growth
- Level: Intermediate
- Last reviewed: 2026-10-05
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/sample-size/>)

## What it means

Sample size is the count of units—such as users, accounts, or clusters—contributing to an analysis. In an experiment it should be planned from the decision-relevant effect, desired power or precision, error rate, variance, allocation, and design rather than chosen only by calendar time.

## Why it matters

Too few independent units produce wide uncertainty and missed effects; unnecessary exposure wastes time and can increase risk. Counting events instead of the randomization unit can create false precision.

## When to use it

Plan sample size before collection for experiments, surveys, and measurement studies. Account for clustering, attrition, eligibility, and repeated observations; use an appropriate sequential design when decisions truly require ongoing monitoring.

## How it works

1. Identify the independent unit and primary outcome.
2. Set the practical effect or precision target and acceptable error rates.
3. Estimate baseline rate, variance, allocation, and expected exclusions.
4. Calculate the required eligible units and translate them into duration.
5. Monitor eligibility and data quality without changing the target opportunistically.

## A practical example

A collaboration product randomizes at account level because teammates affect each other. It plans sample size from accounts, not click events, and delays launch until enough accounts can receive a stable experience.

## Where it breaks

Calculations depend on uncertain assumptions and do not make a biased design valid. More data can make trivial effects detectable, while complex dependence can reduce effective sample size substantially.

## Common mistakes

- Using event count when assignment is by account.
- Stopping when a convenient calendar date arrives despite inadequate exposure.
- Ignoring attrition, clustering, or unequal allocation.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/sample-size/#visualization-sample-size>)

## Sources

1. [Engineering Statistics Handbook](<https://www.itl.nist.gov/div898/handbook/>) — National Institute of Standards and Technology · Government statistical reference
2. [ASA Statement on Statistical Significance and P-Values](<https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf>) — American Statistical Association · Professional society statement

## Related topics

- [Statistical Power](<https://wiki.soroush.uk/topic/statistical-power/>)
- [Minimum Detectable Effect](<https://wiki.soroush.uk/topic/minimum-detectable-effect/>)
- [Randomization](<https://wiki.soroush.uk/topic/randomization/>)
