Explainability
Explainability is the ability to provide people with meaningful, faithful information about how an AI system produces a result or behaves.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/explainability/
01What it means
Explainability concerns explanations of a system's output or operation that people can understand and use. Explainable AI, often abbreviated XAI, includes methods that describe individual predictions and broader model behavior. Interpretability often refers to understanding a model's internal operation directly, although usage varies. Disclosure that AI is involved is transparency; it does not itself explain a decision.[1][2]
- 01Affected personWhat happened, important factors, uncertainty, and how to contest
- 02OperatorEvidence, failure signals, safe action, and escalation
- 03DeveloperModel behavior, data, diagnostics, and reproducible cases
- 04Auditor or regulatorControls, provenance, evaluation, accountability, and impact
- 05Product ownerFitness, risk, tradeoffs, monitoring, and withdrawal criteria
An explanation must be meaningful and faithful enough for its purpose; persuasive detail can increase unjustified confidence without improving understanding.
02Why it matters
Users may need to identify wrong inputs, understand a recommendation, or contest a consequential result. Product teams also need explanations to diagnose shortcuts and failures. An explanation should help a concrete decision, rather than merely make a model seem credible.
03When to use it
Use explanations when they support correction, investigation, learning, or recourse. Choose content and detail for the audience and consequence. A concise source passage or data correction route may be more useful than a technical feature-attribution chart.
04How it works
- Identify who needs an explanation and what they must be able to understand or do with it.
- Choose whether to explain one result, overall behavior, input evidence, or the system's operating limits.
- Select a method appropriate to the model and task, distinguishing direct interpretation from a post-hoc approximation.
- Test whether the explanation faithfully represents the relevant behavior and remains stable under reasonable input changes.
- Evaluate user comprehension and decisions, including incorrect model outputs, and provide correction or escalation routes.
05A practical example
A marketplace explains that a listing's category suggestion depends strongly on its title. A seller notices a misleading phrase and corrects it. The team tests whether this explanation helps users fix categorization errors; an attractive explanation alone is not counted as success.
06Where it breaks
Post-hoc explanations can approximate behavior imperfectly or vary across similar inputs. Feature importance is not proof of causation, and generated rationales can be plausible without reflecting the process that produced an answer. Explanations cannot establish fairness or compensate for an unsuitable decision system.
Common mistakes
- Using a fluent model-generated rationale as a verified account of its internal computation.
- Treating a feature contribution as a causal effect or advice about how to obtain a desired outcome.
- Showing the same technical explanation to every audience.
- Measuring increased trust without checking whether reliance becomes more appropriate.
07Sources
- Four Principles of Explainable Artificial IntelligenceNational Institute of Standards and Technology · Government research reportFour Principles of Explainable Artificial Intelligence
- "Why Should I Trust You?": Explaining the Predictions of Any ClassifierACM SIGKDD · Peer-reviewed research"Why Should I Trust You?": Explaining the Predictions of Any Classifier