Sample Size
Sample size is the number of independent analysis units included in a study or experiment, chosen to support a stated precision or detection goal.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/sample-size/
01What it means
Sample size is the count of units—such as users, accounts, or clusters—contributing to an analysis. In an experiment it should be planned from the decision-relevant effect, desired power or precision, error rate, variance, allocation, and design rather than chosen only by calendar time.[1][2]
- Independent unit
- Users, accounts, clusters, sessions
- Primary outcome
- Metric and analysis method
- Effect or precision
- Decision-relevant planning target
- Error rates
- Power and false-positive tolerance
- Baseline variability
- Rate, variance, clustering
- Feasibility
- Allocation, exclusions, traffic, duration
Translate eligible units into duration only after accounting for allocation, exclusions, traffic, clusters, and the independent analysis unit.
02Why it matters
Too few independent units produce wide uncertainty and missed effects; unnecessary exposure wastes time and can increase risk. Counting events instead of the randomization unit can create false precision.
03When to use it
Plan sample size before collection for experiments, surveys, and measurement studies. Account for clustering, attrition, eligibility, and repeated observations; use an appropriate sequential design when decisions truly require ongoing monitoring.
04How it works
- Identify the independent unit and primary outcome.
- Set the practical effect or precision target and acceptable error rates.
- Estimate baseline rate, variance, allocation, and expected exclusions.
- Calculate the required eligible units and translate them into duration.
- Monitor eligibility and data quality without changing the target opportunistically.
05A practical example
A collaboration product randomizes at account level because teammates affect each other. It plans sample size from accounts, not click events, and delays launch until enough accounts can receive a stable experience.
06Where it breaks
Calculations depend on uncertain assumptions and do not make a biased design valid. More data can make trivial effects detectable, while complex dependence can reduce effective sample size substantially.
Common mistakes
- Using event count when assignment is by account.
- Stopping when a convenient calendar date arrives despite inadequate exposure.
- Ignoring attrition, clustering, or unequal allocation.
07Sources
- Engineering Statistics HandbookNational Institute of Standards and Technology · Government statistical referenceEngineering Statistics Handbook
- ASA Statement on Statistical Significance and P-ValuesAmerican Statistical Association · Professional society statementASA Statement on Statistical Significance and P-Values