Automation Bias
Automation bias is the tendency to rely on automated advice in ways that cause people to miss problems or accept incorrect recommendations.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/automation-bias/
01What it means
Automation bias occurs when automated output displaces appropriate attention to other evidence. It can cause commission errors, such as following a wrong recommendation, and omission errors, such as overlooking a problem because the system did not flag it. It predates generative AI. Reliance is not inherently biased: the concern is reliance that exceeds what the system's reliability and task context justify.[1][2]
Test wrong and incomplete advice—not only overall task speed
Commission errors follow incorrect advice; omission errors miss problems the automation did not flag. Expertise, incentives, workload, and interface design all affect the pathway.
02Why it matters
A human approval step can fail even when reviewers are competent. Defaults, time pressure, workload, and apparently authoritative advice influence whether people verify a result. Teams should measure the combined human-system outcome, particularly when the automation is wrong.
03When to use it
Consider automation bias in recommendation, moderation, ranking, decision-support, and approval workflows. Test with realistic workloads and occasional incorrect suggestions. Avoid adding friction indiscriminately: extra verification should improve decisions enough to justify its effort.
04How it works
- Identify where automated suggestions influence human decisions and which independent evidence is available.
- Observe decisions with and without advice, including controlled cases where the advice is wrong or incomplete.
- Measure acceptance of incorrect suggestions and missed issues separately from overall completion speed.
- Test interventions such as independent initial judgment, accessible source evidence, reversible actions, and selective review prompts.
- Reassess error detection, workload, accessibility, and appropriate reliance over repeated use.
05A practical example
A moderation queue preselects an AI recommendation. In a controlled study, reviewers accept several incorrect removals despite contradictory context. The team tests requiring an initial judgment for ambiguous cases and compares corrected errors and review time before changing the workflow.
06Where it breaks
Observed reliance also reflects expertise, incentives, task difficulty, and system quality. Findings from one setting may not transfer to another. Cognitive forcing interventions can slow work or frustrate users, and explanations sometimes encourage unwarranted confidence rather than better judgment.
Common mistakes
- Blaming reviewers without examining workload, defaults, and evidence access.
- Treating every accepted recommendation as successful assistance.
- Adding a generic warning while leaving the decision environment unchanged.
- Assuming explanations or a confirmation button eliminate overreliance.
07Sources
- Decision Making In A High-Tech World: Automation Bias and CountermeasuresNASA Technical Reports Server · Research reportDecision Making In A High-Tech World: Automation Bias and Countermeasures
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingACM · Peer-reviewed researchTo Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making