frameworkResearch & behavior

Research Validity and Reliability

Criteria for judging whether research supports the intended conclusion and whether its measurement or procedure is sufficiently consistent.

Download Markdown (.md)

01What it means

Validity asks whether evidence supports the interpretation or decision a study claims to support; reliability asks whether measurement or procedure is consistent enough for its purpose. Internal validity concerns whether the study design supports the proposed explanation, while external validity concerns how well findings transfer to the people, settings, tasks, and times of interest.[1][2]

Interpretation, transfer, and measurement consistency Evidence quality depends on the claim it is intended to support
QuestionValidityReliability
MeaningDoes evidence support the claimed interpretation?Is measurement consistent enough for purpose?
Within studyConfounds, selection, demand, alternativesItems, raters, sessions, repeated measures
Beyond studyTransfer across people, settings, tasks, timesConsistency under the relevant conditions

Validity and reliability are not universal badges; evaluate them for the construct, population, setting, task, method, and decision.

02Why it matters

A precise, repeatable measure can consistently capture the wrong construct, and a realistic study can contain uncontrolled alternative explanations. Separating these qualities helps teams challenge impressive-looking numbers, choose appropriate methods, and communicate confidence without treating all research as equally generalizable.

03When to use it

Assess validity and reliability when planning a study, reviewing evidence, combining sources, or making consequential claims. Emphasize the dimensions relevant to the method: consistency matters strongly in benchmarking, internal validity in causal tests, and transfer to context in field and product research.

04How it works

  1. State the construct, population, context, and claim the evidence is intended to support.
  2. Check whether tasks, questions, measures, and instruments actually represent that construct.
  3. Identify plausible confounds, demand effects, selection problems, and alternative explanations.
  4. Examine consistency across items, raters, sessions, or repeated measures where appropriate.
  5. Test transfer with realistic contexts, relevant participants, triangulation, or replication, then qualify the claim.

05A practical example

Example

A benchmark records checkout completion with highly consistent task instructions, but participants use dummy cards and face none of the trust or budget constraints of real purchases. The measure may be reliable within the study while its validity for predicting live conversion remains limited.

06Where it breaks

Validity is not a single score and cannot be guaranteed by one checklist, large sample, or statistical threshold. Reliability requirements differ across qualitative interpretation and standardized measurement. Improving control can reduce realism, so evidence quality involves explicit tradeoffs rather than maximizing every dimension at once.

Common mistakes

  • Treating repeatability as proof that a study measures the intended concept.
  • Claiming broad generalization from one narrow population, task, setting, or moment.
  • Using sample size or statistical significance as a substitute for examining design validity.

07Sources

  1. Internal vs. External Validity of UX StudiesNielsen Norman Group · Professional guidanceInternal vs. External Validity of UX Studies
  2. Strength of User Research EvidenceNielsen Norman Group · Professional guidanceStrength of User Research Evidence
View saved topics

Start typing to search titles, definitions, categories, and frameworks.