RedLectern topic
Explanations of outputs · investigation of internal mechanisms · evidence and limitations.
For: Technical learners · researchers
After this page you can: Distinguish a plausible explanation from a tested mechanism.
Page reviewed
Part of: Artificial Intelligence › AI safety and governance
AI interpretability studies what can be learned from a system’s outputs and internal computations. An explanation might summarize which input features influenced one prediction, describe how predictions vary across a dataset, or propose a pattern inside a neural network that contributes to a behavior. These accounts answer different questions. A useful description of an output is not automatically a description of the computation that produced it.
The central habit is to ask what evidence could distinguish a plausible story from a tested mechanism. For an output explanation, check what model, inputs, and comparison the method actually covers. For an internal explanation, ask whether changing the proposed feature or circuit changes the behavior as predicted, and whether the effect persists across relevant tests. An intervention can strengthen a mechanistic claim, but its limits matter too: a feature can have unexpected effects, and a narrow evaluation may not generalize. Treat explanations as claims to evaluate, with scope and uncertainty, rather than as direct access to a model’s full reasoning.
Read the introductory chapters on interpretability, goals, and methods overview. Then compare what an output explanation can show with what an internal-mechanism claim would require.
Start with: Interpretable Machine Learning: A Guide for Making Black Box Models Explainable
Interpretability and explainability are used in overlapping ways, so begin by asking what the explanation is meant to describe. It may make a model’s output easier to understand, or it may describe patterns in the model’s operation. A local account concerns one prediction; a global account summarizes behavior across a dataset. Neither label alone tells you whether the account is faithful to the computation or useful for the question at hand.
Check the claim: Identify the system, output or behavior being explained, and the evidence behind the account. A clear feature attribution can be a useful description without establishing that the named feature is a complete or causal mechanism. Internal mechanisms require evidence about how components contribute to behavior.
Maps approaches including by-design and post-hoc methods, with local and global methods.
Explainability
Providing understandable accounts of a system’s outputs or operation.
Mechanistic interpretability
Investigating internal model mechanisms associated with particular behaviors.
A report says an image classifier relied on a highlighted patch to label one picture. State what the report supports and one additional test needed before claiming a general mechanism.
It supports that the patch was associated with the explanation for that prediction. Test whether manipulating or removing the patch changes predictions across relevant examples, and check whether the explanation remains faithful beyond that one picture.
Output explanations describe a prediction or a model’s behavior over examples. Some methods change input features and observe output changes; others assign contributions to features or approximate a model locally. Global summaries and local explanations answer different questions, and a method’s result depends on its assumptions and setup. For example, permutation feature importance measures how prediction error changes when feature values are shuffled; correlated features can make the resulting data unrealistic.
Practice the distinction: Ask whether an explanation describes the model’s response to a particular input, summarizes a dataset, or supports a causal claim about the world. A feature’s importance to prediction error does not by itself say whether increasing that feature raises the prediction or establish a real-world cause.
Shows how shuffling a feature is used to measure changes in model error and discusses limitations.
Uses examples of models relying on artifacts to motivate checking what a model learned.
A feature-importance report says shuffling feature A raised a classifier’s error. What does that support, and what does it leave unanswered about the effect of A?
It supports that the model’s prediction error changed when A was shuffled in that evaluation. It does not show whether increasing A raises or lowers a prediction, whether the result reflects correlated features, or whether A causes the real-world outcome.
Mechanistic work proposes how internal units, features, or circuits contribute to a behavior. Evidence can begin with patterns in activations or examples that make a feature label plausible. A stronger test intervenes on the proposed component and checks whether outputs change in the predicted way. The intervention should be evaluated across relevant examples and for unintended effects; a few intuitive demonstrations do not establish that the proposed account is complete or reliable.
A useful test sequence: State the proposed mechanism and a prediction it makes. Change or ablate the proposed component, compare outputs with a suitable baseline, and test other behaviors that could reveal off-target effects. Interpret the result within the model, feature, and evaluations studied. A successful intervention is evidence for a contribution under those conditions, not a proof that the label captures every function of the component.
Describes intervention tests and reports both expected and off-target effects, plus evaluation limits.
Introduces a course unit that builds from transformer components toward circuits and features.
A proposed feature is said to cause a model behavior because examples activating it fit the feature’s label. Name one intervention test and one limitation to check.
Change the feature’s activation and compare outputs against an unmodified baseline on relevant examples. Check for off-target behavior and whether the evaluation covers only a narrow set of examples; matching activation contexts alone does not establish the feature’s behavioral effect.
No recent updates are available.
Christoph Molnar · 2025
Compare methods for explaining individual predictions and broader model behavior, and learn to question what those explanations establish. The current edition is available free online under CC BY-NC-SA 4.0.
Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller (eds.) · 2019
This edited collection covers deep-learning attribution, evaluation of explanations, and neural-network reverse engineering. Its 2019 scope predates contemporary LLM mechanistic interpretability, so treat it as background rather than a guide to current LLM circuits.