# AI Hallucination

An AI hallucination is generated content that is unsupported by available evidence, inconsistent with the input, or factually incorrect while appearing plausible.

- Type: term
- Domain: Engineering & architecture
- Level: Foundational
- Last reviewed: 2026-09-26
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/hallucination/>)

## What it means

Hallucination is a reliability failure in which a generative model produces claims or details not grounded in the source, prompt, or verifiable reality. It can be intrinsic, contradicting supplied material, or extrinsic, adding unsupported information. The term describes an output problem, not intent or a human-like mental state.

## Why it matters

Plausible fabrication can mislead users, corrupt downstream data, and create legal, financial, safety, or reputational harm. Because confidence and writing quality do not reliably reveal correctness, product design must make verification practical and reduce the authority granted to generated text.

## When to use it

Track hallucination wherever generated claims, summaries, citations, or extracted facts affect decisions. Define evidence requirements and severity by use case; creative ideation may tolerate invention, while policy, health, finance, and automated actions require much stronger controls or a different approach.

## How it works

1. Define what counts as supported, contradicted, or unverifiable for the specific task.
2. Create representative examples with source evidence and expert adjudication where needed.
3. Measure claim-level factuality and the severity of errors, not only general response quality.
4. Reduce risk through grounding, constrained tasks, citations, validation, abstention, and human review.
5. Monitor production failures and update evidence, evaluations, and workflows as conditions change.

## A practical example

A customer-support model invents a refund exception that sounds consistent with company policy. The team changes the workflow to retrieve approved clauses, require clause IDs for policy claims, block uncited recommendations, and route ambiguous cases to an agent.

## Where it breaks

No single metric captures all factual errors, and automated judges can share model biases. Grounding can retrieve wrong or stale material, citations may not support the claim, and human reviewers may trust fluent output. Some apparent hallucinations reflect ambiguous or conflicting sources.

## Common mistakes

- Using hallucination to describe every undesirable output, including harmful but evidence-supported content.
- Asking the same model whether its own answer is true and treating that as verification.
- Adding citations without checking that each cited passage entails the associated claim.
- Reporting an average error rate without distinguishing consequential failures.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/hallucination/#visualization-hallucination>)

## Sources

1. [Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile](<https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf>) — National Institute of Standards and Technology · Government framework
2. [Survey of Hallucination in Natural Language Generation](<https://dl.acm.org/doi/10.1145/3571730>) — ACM Computing Surveys · Peer-reviewed research

## Related topics

- [Grounding](<https://wiki.soroush.uk/topic/grounding/>)
- [Retrieval-Augmented Generation \(RAG\)](<https://wiki.soroush.uk/topic/retrieval-augmented-generation/>)
- [Model Evaluation](<https://wiki.soroush.uk/topic/model-evaluation/>)
