Human in the Loop (HITL)
Human in the loop assigns people a defined role in reviewing, correcting, approving, or interrupting an AI system at meaningful points in its operation.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/human-in-the-loop/
01What it means
Human in the loop (HITL) is a system design in which human input is deliberately incorporated during training, inference, or action. In products it often means review, exception handling, confirmation, correction, or override. A person merely present near automation is not meaningful oversight; the reviewer needs information, time, competence, authority, and a usable alternative.[1][2]
- 01System proposesPrediction, content, action
- 02Route for reviewThreshold or exception
- 03Human decidesConfirm, correct, override
- 04Action + recordPreserve rationale
- 05EvaluateQuality, workload, delay, bias
Reviewers need evidence, time, competence, authority, and an alternative.
02Why it matters
Human involvement can catch context-specific errors, handle exceptions, protect autonomy, and create feedback for improvement. Poorly designed review can instead transfer hidden labor and liability to users while automation bias, fatigue, and throughput pressure make intervention unlikely.
03When to use it
Use HITL when uncertainty or consequence makes judgment valuable and reviewers can act effectively. Automate low-risk, well-understood cases and route defined exceptions. Do not use a confirmation dialog as justification for unsafe automation whose output a person cannot realistically verify.
04How it works
- Identify decisions or actions where human judgment changes risk, quality, rights, or accountability.
- Define routing thresholds, reviewer qualifications, evidence, time, authority, and service expectations.
- Design review interfaces that show source material, uncertainty, changes, consequences, and alternatives.
- Record corrections and overrides without treating every human response as clean training truth.
- Measure error detection, reviewer workload, disagreement, delay, automation bias, and user outcomes.
05A practical example
An expense system auto-codes routine receipts but routes unusual tax categories to trained reviewers. The interface highlights extracted fields against the image, reviewers can correct or reject the suggestion, and managers track false approvals and review burden—not just automation rate.
06Where it breaks
People make errors, disagree, and can become over-reliant on system suggestions. Review adds delay and cost and may expose sensitive data. At high volume, nominal oversight becomes a rubber stamp; feedback loops can also encode reviewer bias into future models.
Common mistakes
- Calling any user confirmation meaningful human oversight.
- Showing a recommendation without the evidence and uncertainty needed to assess it.
- Optimizing automation rate while ignoring reviewer workload and missed harmful errors.
- Feeding every correction back into training without governance or quality checks.
07Sources
- Artificial Intelligence Risk Management Framework (AI RMF 1.0)National Institute of Standards and Technology · Government frameworkArtificial Intelligence Risk Management Framework (AI RMF 1.0)
- People + AI GuidebookGoogle People + AI Research · Design guidePeople + AI Guidebook