# Data Labeling

Data labeling assigns task-specific annotations to examples so models can be trained or evaluated against defined expectations.

- Type: practice
- Domain: Engineering & architecture
- Level: Intermediate
- Last reviewed: 2026-10-06
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/data-labeling/>)

## What it means

Data labeling, also called data annotation, adds categories, spans, boundaries, ratings, or other target information to examples. Humans, automated systems, or combined workflows may produce the labels. A reference set is often called ground truth, but labels can contain mistakes, ambiguity, and differences in judgment. This practice is distinct from naming navigation items in an interface or documenting a dataset.

## Why it matters

Labels turn a product's quality or policy expectations into training and evaluation targets. If annotators interpret a category differently, the model may learn an inconsistent task and its measured quality may be misleading. Labeling effort therefore includes specification and quality review, not only throughput.

## When to use it

Use labeling for supervised learning, retrieval relevance judgments, preference comparisons, and evaluation rubrics. Pilot the task before scaling it. When the intended judgment is unclear or people legitimately disagree, refine the product decision or preserve that variation rather than forcing an apparently definitive answer.

## How it works

1. Define the unit to label, intended use, category boundaries, and options for uncertainty or insufficient context.
2. Write instructions with clear examples and recruit annotators with suitable knowledge of the task and population.
3. Pilot independent labeling on overlapping examples and inspect disagreement patterns.
4. Correct procedural errors, clarify ambiguous guidance, and document how substantive disagreement is handled.
5. Audit quality throughout production and version the labels, instructions, and provenance with the dataset.

## A practical example

A feedback classifier distinguishes bug reports from usability problems. During a pilot, reviewers disagree about users failing to discover an existing feature. The team clarifies the intended product categories, keeps multi-issue feedback eligible for multiple labels, and reviews a fresh sample before commissioning the full dataset.

## Where it breaks

Agreement measures consistency rather than objective truth. Annotators can share biases, and subjective tasks may have several defensible judgments. Time pressure, missing context, and fatigue affect quality. Automated labeling can scale a mistaken rule or model bias as easily as a correct one.

## Common mistakes

- Calling labels ground truth without examining how they were produced.
- Choosing majority vote automatically when disagreement reflects meaningful perspectives.
- Changing instructions midway without recording affected examples.
- Optimizing cost per label while ignoring ambiguity, expertise, and audit effort.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/data-labeling/#visualization-data-labeling>)

## Sources

1. [Training data labeling using humans with Amazon SageMaker Ground Truth](<https://docs.aws.amazon.com/sagemaker/latest/dg/sms.html>) — Amazon Web Services · Official documentation
2. [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](<https://aclanthology.org/2022.tacl-1.6/>) — Transactions of the Association for Computational Linguistics · Peer-reviewed research

## Related topics

- [Dataset Card](<https://wiki.soroush.uk/topic/dataset-card/>)
- [Model Evaluation](<https://wiki.soroush.uk/topic/model-evaluation/>)
- [Algorithmic Fairness](<https://wiki.soroush.uk/topic/algorithmic-fairness/>)
