AI Hallucination
An AI hallucination is generated content that is unsupported by available evidence, inconsistent with the input, or factually incorrect while appearing plausible.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/hallucination/
01What it means
Hallucination is a reliability failure in which a generative model produces claims or details not grounded in the source, prompt, or verifiable reality. It can be intrinsic, contradicting supplied material, or extrinsic, adding unsupported information. The term describes an output problem, not intent or a human-like mental state.[1][2]
- 01Task + contextMay be incomplete, ambiguous, outdated, or adversarial
- 02Model generationProduces likely continuation rather than guaranteed truth
- 03Plausible claimUnsupported, inconsistent, or factually incorrect content
- 04Product presentationConfidence cues may conceal uncertainty and provenance
- 05Human or system useUnverified content can influence a decision or action
- 06Verification boundaryGround, cite, check, constrain, or refuse before harm
Hallucination is not a single model defect: missing evidence, ambiguous input, retrieval failure, incentives, and product presentation all shape whether unsupported content causes harm.
02Why it matters
Plausible fabrication can mislead users, corrupt downstream data, and create legal, financial, safety, or reputational harm. Because confidence and writing quality do not reliably reveal correctness, product design must make verification practical and reduce the authority granted to generated text.
03When to use it
Track hallucination wherever generated claims, summaries, citations, or extracted facts affect decisions. Define evidence requirements and severity by use case; creative ideation may tolerate invention, while policy, health, finance, and automated actions require much stronger controls or a different approach.
04How it works
- Define what counts as supported, contradicted, or unverifiable for the specific task.
- Create representative examples with source evidence and expert adjudication where needed.
- Measure claim-level factuality and the severity of errors, not only general response quality.
- Reduce risk through grounding, constrained tasks, citations, validation, abstention, and human review.
- Monitor production failures and update evidence, evaluations, and workflows as conditions change.
05A practical example
A customer-support model invents a refund exception that sounds consistent with company policy. The team changes the workflow to retrieve approved clauses, require clause IDs for policy claims, block uncited recommendations, and route ambiguous cases to an agent.
06Where it breaks
No single metric captures all factual errors, and automated judges can share model biases. Grounding can retrieve wrong or stale material, citations may not support the claim, and human reviewers may trust fluent output. Some apparent hallucinations reflect ambiguous or conflicting sources.
Common mistakes
- Using hallucination to describe every undesirable output, including harmful but evidence-supported content.
- Asking the same model whether its own answer is true and treating that as verification.
- Adding citations without checking that each cited passage entails the associated claim.
- Reporting an average error rate without distinguishing consequential failures.
07Sources
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence ProfileNational Institute of Standards and Technology · Government frameworkArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys · Peer-reviewed researchSurvey of Hallucination in Natural Language Generation