Data Labeling
Data labeling assigns task-specific annotations to examples so models can be trained or evaluated against defined expectations.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/data-labeling/
01What it means
Data labeling, also called data annotation, adds categories, spans, boundaries, ratings, or other target information to examples. Humans, automated systems, or combined workflows may produce the labels. A reference set is often called ground truth, but labels can contain mistakes, ambiguity, and differences in judgment. This practice is distinct from naming navigation items in an interface or documenting a dataset.[1][2]
- 01Define task + policyClasses, boundaries, uncertainty, and exclusions
- 02Sample examplesRepresentative, difficult, and safety-relevant cases
- 03AnnotateTrained people, tools, or assisted workflows
- 04Resolve disagreementAdjudication, rationale, and guideline revision
- 05Audit datasetQuality, coverage, bias, privacy, and provenance
- 06Version + monitorTrack policy and distribution changes
Labels encode a task and policy rather than objective truth; ambiguity, worker conditions, representation, privacy, and drift affect model behavior.
02Why it matters
Labels turn a product's quality or policy expectations into training and evaluation targets. If annotators interpret a category differently, the model may learn an inconsistent task and its measured quality may be misleading. Labeling effort therefore includes specification and quality review, not only throughput.
03When to use it
Use labeling for supervised learning, retrieval relevance judgments, preference comparisons, and evaluation rubrics. Pilot the task before scaling it. When the intended judgment is unclear or people legitimately disagree, refine the product decision or preserve that variation rather than forcing an apparently definitive answer.
04How it works
- Define the unit to label, intended use, category boundaries, and options for uncertainty or insufficient context.
- Write instructions with clear examples and recruit annotators with suitable knowledge of the task and population.
- Pilot independent labeling on overlapping examples and inspect disagreement patterns.
- Correct procedural errors, clarify ambiguous guidance, and document how substantive disagreement is handled.
- Audit quality throughout production and version the labels, instructions, and provenance with the dataset.
05A practical example
A feedback classifier distinguishes bug reports from usability problems. During a pilot, reviewers disagree about users failing to discover an existing feature. The team clarifies the intended product categories, keeps multi-issue feedback eligible for multiple labels, and reviews a fresh sample before commissioning the full dataset.
06Where it breaks
Agreement measures consistency rather than objective truth. Annotators can share biases, and subjective tasks may have several defensible judgments. Time pressure, missing context, and fatigue affect quality. Automated labeling can scale a mistaken rule or model bias as easily as a correct one.
Common mistakes
- Calling labels ground truth without examining how they were produced.
- Choosing majority vote automatically when disagreement reflects meaningful perspectives.
- Changing instructions midway without recording affected examples.
- Optimizing cost per label while ignoring ambiguity, expertise, and audit effort.
07Sources
- Training data labeling using humans with Amazon SageMaker Ground TruthAmazon Web Services · Official documentationTraining data labeling using humans with Amazon SageMaker Ground Truth
- Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective AnnotationsTransactions of the Association for Computational Linguistics · Peer-reviewed researchDealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations