Survey Design
The practice of creating questionnaires and sampling plans that measure attitudes, experiences, or characteristics with controlled sources of error.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/survey-design/
01What it means
Survey design turns a research question into a structured instrument, a sampling approach, and an analysis plan. It includes deciding whom to invite, writing questions that respondents interpret consistently, choosing response options, ordering items, testing the questionnaire, and managing delivery. A polished form is not enough: coverage, nonresponse, measurement, and analysis errors can all distort results.[1][2]
- 01Construct + decisionPopulation and analysis plan
- 02Sample + recruitWho can and cannot be reached
- 03Write questionsNeutral, single-focus, suitable recall
- 04Design responses + orderExhaustive, non-overlapping, accessible
- 05Cognitive test + pilotInspect quality, uncertainty, limitations
A polished form cannot correct coverage, nonresponse, measurement, or analysis errors created by the study design.
02Why it matters
Surveys can estimate how common an experience or attitude is across a defined population, but small wording choices can produce large differences. Careful design makes results interpretable and prevents teams from presenting a self-selected pulse check as precise customer truth. It also reduces respondent burden and protects people from unnecessary or intrusive collection.
03When to use it
Use surveys when the team needs comparable responses from many people, wants to quantify themes discovered qualitatively, track a measure over time, or segment experiences. Use another method when the question is exploratory, requires observation, or depends on nuanced reasoning that fixed responses cannot capture.
04How it works
- Define the construct, target population, decision to be informed, and analysis plan before drafting questions.
- Choose a sampling and recruitment approach, documenting who can and cannot be reached.
- Write neutral, single-focus questions with exhaustive, non-overlapping response options and suitable recall periods.
- Order and format the questionnaire to reduce priming, fatigue, accessibility barriers, and accidental disclosure.
- Cognitively test and pilot the survey, then inspect response quality and report uncertainty and limitations.
05A practical example
After interviews suggest that freelancers miss invoice reminders, a finance product surveys active customers about incidents in the previous 30 days. The questionnaire separates “did not see,” “saw too late,” and “saw but postponed,” randomizes non-ordinal options, and records work type. Results show the issue concentrates among multi-client users, guiding targeted reminder controls.
06Where it breaks
Surveys measure what respondents understand, remember, and choose to disclose. Poor samples, low response, leading wording, order effects, and satisficing can overwhelm apparent precision. Correlation does not establish cause. Very long questionnaires degrade quality, and percentages without denominators or uncertainty can mislead. Pretesting and transparent reporting are essential.
Common mistakes
- Writing double-barreled questions such as asking whether a feature is “fast and easy.”
- Using only convenient respondents while describing results as representative of all users.
- Starting collection before defining how each answer will support a decision or analysis.
07Sources
- Writing Survey QuestionsPew Research Center · Research methods guidanceWriting Survey Questions
- Questionnaire Design and TranslationPew Research Center · Research methods guidanceQuestionnaire Design and Translation