metricData, experimentation & growth

Sample Size

Sample size is the number of independent analysis units included in a study or experiment, chosen to support a stated precision or detection goal.

Download Markdown (.md)

01What it means

Sample size is the count of units—such as users, accounts, or clusters—contributing to an analysis. In an experiment it should be planned from the decision-relevant effect, desired power or precision, error rate, variance, allocation, and design rather than chosen only by calendar time.[1][2]

Planned evidence volume Required units follow from the decision, design, variability, and error tolerance
Required sample size
Independent unit
Users, accounts, clusters, sessions
Primary outcome
Metric and analysis method
Effect or precision
Decision-relevant planning target
Error rates
Power and false-positive tolerance
Baseline variability
Rate, variance, clustering
Feasibility
Allocation, exclusions, traffic, duration

Translate eligible units into duration only after accounting for allocation, exclusions, traffic, clusters, and the independent analysis unit.

02Why it matters

Too few independent units produce wide uncertainty and missed effects; unnecessary exposure wastes time and can increase risk. Counting events instead of the randomization unit can create false precision.

03When to use it

Plan sample size before collection for experiments, surveys, and measurement studies. Account for clustering, attrition, eligibility, and repeated observations; use an appropriate sequential design when decisions truly require ongoing monitoring.

04How it works

  1. Identify the independent unit and primary outcome.
  2. Set the practical effect or precision target and acceptable error rates.
  3. Estimate baseline rate, variance, allocation, and expected exclusions.
  4. Calculate the required eligible units and translate them into duration.
  5. Monitor eligibility and data quality without changing the target opportunistically.

05A practical example

Example

A collaboration product randomizes at account level because teammates affect each other. It plans sample size from accounts, not click events, and delays launch until enough accounts can receive a stable experience.

06Where it breaks

Calculations depend on uncertain assumptions and do not make a biased design valid. More data can make trivial effects detectable, while complex dependence can reduce effective sample size substantially.

Common mistakes

  • Using event count when assignment is by account.
  • Stopping when a convenient calendar date arrives despite inadequate exposure.
  • Ignoring attrition, clustering, or unequal allocation.

07Sources

  1. Engineering Statistics HandbookNational Institute of Standards and Technology · Government statistical referenceEngineering Statistics Handbook
  2. ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values
View saved topics

Start typing to search titles, definitions, categories, and frameworks.