# Explainability

Explainability is the ability to provide people with meaningful, faithful information about how an AI system produces a result or behaves.

- Type: principle
- Domain: Data, experimentation & growth
- Level: Advanced
- Last reviewed: 2026-09-26
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/explainability/>)

## What it means

Explainability concerns explanations of a system's output or operation that people can understand and use. Explainable AI, often abbreviated XAI, includes methods that describe individual predictions and broader model behavior. Interpretability often refers to understanding a model's internal operation directly, although usage varies. Disclosure that AI is involved is transparency; it does not itself explain a decision.

## Why it matters

Users may need to identify wrong inputs, understand a recommendation, or contest a consequential result. Product teams also need explanations to diagnose shortcuts and failures. An explanation should help a concrete decision, rather than merely make a model seem credible.

## When to use it

Use explanations when they support correction, investigation, learning, or recourse. Choose content and detail for the audience and consequence. A concise source passage or data correction route may be more useful than a technical feature-attribution chart.

## How it works

1. Identify who needs an explanation and what they must be able to understand or do with it.
2. Choose whether to explain one result, overall behavior, input evidence, or the system's operating limits.
3. Select a method appropriate to the model and task, distinguishing direct interpretation from a post-hoc approximation.
4. Test whether the explanation faithfully represents the relevant behavior and remains stable under reasonable input changes.
5. Evaluate user comprehension and decisions, including incorrect model outputs, and provide correction or escalation routes.

## A practical example

A marketplace explains that a listing's category suggestion depends strongly on its title. A seller notices a misleading phrase and corrects it. The team tests whether this explanation helps users fix categorization errors; an attractive explanation alone is not counted as success.

## Where it breaks

Post-hoc explanations can approximate behavior imperfectly or vary across similar inputs. Feature importance is not proof of causation, and generated rationales can be plausible without reflecting the process that produced an answer. Explanations cannot establish fairness or compensate for an unsuitable decision system.

## Common mistakes

- Using a fluent model-generated rationale as a verified account of its internal computation.
- Treating a feature contribution as a causal effect or advice about how to obtain a desired outcome.
- Showing the same technical explanation to every audience.
- Measuring increased trust without checking whether reliance becomes more appropriate.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/explainability/#visualization-explainability>)

## Sources

1. [Four Principles of Explainable Artificial Intelligence](<https://www.nist.gov/publications/four-principles-explainable-artificial-intelligence>) — National Institute of Standards and Technology · Government research report
2. ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](<https://arxiv.org/abs/1602.04938>) — ACM SIGKDD · Peer-reviewed research

## Related topics

- [Human–AI Interaction](<https://wiki.soroush.uk/topic/human-ai-interaction/>)
- [Model Evaluation](<https://wiki.soroush.uk/topic/model-evaluation/>)
- [Causal Inference](<https://wiki.soroush.uk/topic/causal-inference/>)
