# Research Validity and Reliability

Criteria for judging whether research supports the intended conclusion and whether its measurement or procedure is sufficiently consistent.

- Type: framework
- Domain: Research & behavior
- Level: Advanced
- Last reviewed: 2026-09-25
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/research-validity-and-reliability/>)

## What it means

Validity asks whether evidence supports the interpretation or decision a study claims to support; reliability asks whether measurement or procedure is consistent enough for its purpose. Internal validity concerns whether the study design supports the proposed explanation, while external validity concerns how well findings transfer to the people, settings, tasks, and times of interest.

## Why it matters

A precise, repeatable measure can consistently capture the wrong construct, and a realistic study can contain uncontrolled alternative explanations. Separating these qualities helps teams challenge impressive-looking numbers, choose appropriate methods, and communicate confidence without treating all research as equally generalizable.

## When to use it

Assess validity and reliability when planning a study, reviewing evidence, combining sources, or making consequential claims. Emphasize the dimensions relevant to the method: consistency matters strongly in benchmarking, internal validity in causal tests, and transfer to context in field and product research.

## How it works

1. State the construct, population, context, and claim the evidence is intended to support.
2. Check whether tasks, questions, measures, and instruments actually represent that construct.
3. Identify plausible confounds, demand effects, selection problems, and alternative explanations.
4. Examine consistency across items, raters, sessions, or repeated measures where appropriate.
5. Test transfer with realistic contexts, relevant participants, triangulation, or replication, then qualify the claim.

## A practical example

A benchmark records checkout completion with highly consistent task instructions, but participants use dummy cards and face none of the trust or budget constraints of real purchases. The measure may be reliable within the study while its validity for predicting live conversion remains limited.

## Where it breaks

Validity is not a single score and cannot be guaranteed by one checklist, large sample, or statistical threshold. Reliability requirements differ across qualitative interpretation and standardized measurement. Improving control can reduce realism, so evidence quality involves explicit tradeoffs rather than maximizing every dimension at once.

## Common mistakes

- Treating repeatability as proof that a study measures the intended concept.
- Claiming broad generalization from one narrow population, task, setting, or moment.
- Using sample size or statistical significance as a substitute for examining design validity.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/research-validity-and-reliability/#visualization-research-validity-and-reliability>)

## Sources

1. [Internal vs. External Validity of UX Studies](<https://www.nngroup.com/articles/internal-vs-external-validity/>) — Nielsen Norman Group · Professional guidance
2. [Strength of User Research Evidence](<https://www.nngroup.com/articles/ux-evidence/>) — Nielsen Norman Group · Professional guidance

## Related topics

- [Triangulation](<https://wiki.soroush.uk/topic/triangulation/>)
- [Qualitative and Quantitative Research](<https://wiki.soroush.uk/topic/qualitative-and-quantitative-research/>)
- [User Research Plan](<https://wiki.soroush.uk/topic/user-research-plan/>)
