Sampling in User Research
The deliberate selection of participants or cases whose evidence can support a research question within stated boundaries.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/sampling-in-user-research/
01What it means
Sampling in user research defines the population or context of interest and how participants, cases, events, or records are selected from it. Probability sampling gives eligible units a known chance of selection and supports statistical inference when implemented well. Nonprobability approaches such as purposive, quota, and convenience sampling serve different aims but require narrower claims.[1][2]
- Probability sampling
- Eligible units have a known selection chanceSupports statistical inference when coverage and implementation are sound
- Nonprobability sampling
- Purposive, quota, convenience, or related selectionServes targeted learning with narrower claims
Every approach must report who was included, who was not reached, practical exclusions, and how selection limits the conclusion.
02Why it matters
Findings depend on who and what was available to produce the evidence. A large convenient sample can systematically exclude important users, while a small purposive sample can be appropriate for understanding a specific experience. An explicit sampling rationale prevents headcount from being mistaken for representativeness.
03When to use it
Make sampling decisions whenever research uses a subset of a wider population, whether for interviews, usability studies, surveys, logs, or field sites. Match the approach to whether the goal is depth, variation, issue discovery, comparison, estimation, or generalization.
04How it works
- Define the target population, unit of analysis, and claim the study needs to support.
- Identify characteristics, contexts, or sources of variation relevant to the research question.
- Choose probability or nonprobability sampling and document why it fits the intended inference.
- Recruit or select cases consistently while monitoring coverage, refusals, and practical exclusions.
- Report who was included, who was not reached, and how selection limits the conclusion.
05A practical example
A team studying expense approvals purposively recruits requesters, managers, and finance reviewers from small and large organizations to understand workflow variation. It does not report the proportion of all customers with each problem; a later probability survey estimates prevalence across the customer base.
06Where it breaks
No sampling strategy repairs a poorly defined population or weak measurement. Probability samples can still suffer nonresponse and coverage error, while purposive samples depend on informed selection and may miss unexpected variation. Demographic quotas alone do not make a sample random or substantively representative.
Common mistakes
- Using whoever is easiest to reach while describing them as representative users.
- Choosing a sample size before defining the decision, variation, or intended inference.
- Generalizing percentages from purposive or convenience samples to the full customer population.
07Sources
- Convenience vs. Probability Sampling in UX ResearchNielsen Norman Group · Professional guidanceConvenience vs. Probability Sampling in UX Research
- Purposeful sampling for qualitative data collection and analysis in mixed method implementation researchAdministration and Policy in Mental Health and Mental Health Services Research · Peer-reviewed researchPurposeful sampling for qualitative data collection and analysis in mixed method implementation research