Interpretability: probes, attribution and features

Evaluations tell us what a model does on the inputs we test; interpretability aims to tell us how and why it does it, by examining the model's internal representations and computations. For safety, this matters because behaviour on tests may not reveal a model's dispositions in untested situations, because a model could in principle behave well while being evaluated and differently otherwise, and because understanding the mechanism behind a failure is the best way to fix it. This lesson introduces the main families of interpretability methods that treat the model somewhat as a black box or probe its representations: attribution methods, linear probes and their pitfalls, feature visualization, and representation steering. Lesson 7 turns to mechanistic interpretability, which aims to reverse-engineer the computations themselves.

Goals and kinds of interpretability

Interpretability methods differ in what they explain. Local explanations account for a single prediction: which parts of this input led to this output? Global explanations describe a model's general behaviour or the concepts it represents. Some methods are post hoc, applied to a trained model; others build interpretability into the model's design, such as small decision trees or models constrained to use human-understandable concepts. For large neural networks, post hoc methods dominate. A central requirement is faithfulness: an explanation should reflect what the model actually computes, not merely sound plausible. Plausible but unfaithful explanations can be worse than none, because they create false confidence.

Attribution methods

Attribution methods assign each input feature (a pixel, a word, a token) a score for its contribution to an output. Common approaches include:

• Gradient-based saliency: the gradient of the output with respect to each input feature, showing how sensitive the output is to small changes. Simple but noisy, and it reflects local sensitivity rather than contribution.
• Gradient × input and integrated gradients: integrated gradients, proposed by Sundararajan and colleagues (2017), accumulate gradients along a path from a baseline input (such as a blank image) to the actual input, satisfying desirable axioms such as completeness (attributions sum to the difference in output between input and baseline).
• Perturbation and occlusion: remove or mask parts of the input and observe the change in output, which is intuitive but expensive and sensitive to how masking is done.
• Shapley-value methods such as SHAP: assign contributions based on cooperative game theory, averaging a feature's marginal effect over subsets of other features; widely used for tabular models.
• Attention weights: in transformers, attention patterns show which tokens a head attends to, but attention alone is not a reliable explanation of the output, because information is transformed in many other ways.

Studies have shown that some attribution methods produce similar-looking maps even for randomized models, a warning that visually appealing explanations must be validated with sanity checks.

Linear probes

A probe is a simple classifier, usually linear, trained to predict some property from a model's internal activations. If a linear probe can predict whether a sentence is true, what the sentiment is, or which part of speech a word is, the property is said to be linearly represented at that layer. Probing has revealed that language models represent many concepts linearly, including truthfulness, spatial and temporal information, and the state of board games in models trained only on move sequences.

Probes must be interpreted carefully. A probe that succeeds shows that information is recoverable, not that the model uses it. Because powerful probes can learn the task themselves, Hewitt and Liang proposed control tasks: probing for a meaningless property that is a fixed function of the input, to measure how much of a probe's accuracy reflects the representation versus the probe's own capacity. Causal tests, which intervene on the representation and check whether behaviour changes (Lesson 7), are needed to show that a representation matters.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.neural_network import MLPClassifier
from sklearn.linear_model import LogisticRegression

# Linear probes on the hidden layer of a small network trained to classify digits
digits = load_digits()
X, y = digits.data / 16.0, digits.target
Xtr, ytr, Xte, yte = X[:1300], y[:1300], X[1300:], y[1300:]
net = MLPClassifier(hidden_layer_sizes=(32,), max_iter=2000, random_state=0).fit(Xtr, ytr)
print(f"Network accuracy on digits: {net.score(Xte, yte):.3f}")

def hidden(X):
    return np.maximum(0, X @ net.coefs_[0] + net.intercepts_[0])     # ReLU activations

rng = np.random.default_rng(0)
control_map = rng.integers(0, 2, 10)                                 # random property of each digit
concepts = {"digit is even": (y % 2 == 0), "digit is 5 or more": (y >= 5),
            "control: random digit property": control_map[y] == 1}
for name, lab in concepts.items():
    lab = lab.astype(int)
    on_hidden = LogisticRegression(max_iter=3000).fit(hidden(Xtr), lab[:1300]).score(hidden(Xte), lab[1300:])
    on_pixels = LogisticRegression(max_iter=3000).fit(Xtr, lab[:1300]).score(Xte, lab[1300:])
    print(f"Probe for {name:<32} hidden layer {on_hidden:.3f}   raw pixels {on_pixels:.3f}")

Network accuracy on digits: 0.932
Probe for digit is even                    hidden layer 0.915   raw pixels 0.901
Probe for digit is 5 or more               hidden layer 0.869   raw pixels 0.853
Probe for control: random digit property   hidden layer 0.924   raw pixels 0.907

The network's hidden layer clearly encodes digit identity, so probes for "even" and "5 or more" succeed. But a probe for a random property, assigning each digit an arbitrary label, succeeds just as well. Any property that is a function of the digit is linearly decodable once the digit is represented. Probe accuracy alone therefore says little about whether the network has a specific "evenness" representation, and comparisons with control tasks and with raw inputs are needed before drawing conclusions.

Feature visualization

Feature visualization asks what input most strongly activates a given neuron, channel or direction in a network, by optimizing an input to maximize the activation (with regularization to keep it natural-looking) or by finding the real dataset examples that activate it most. For vision models, this revealed neurons responding to curves, textures, dog faces and, in multimodal models, abstract concepts such as a particular person or emotion across images and text. Visualizations are evocative but must be validated, because neurons are often polysemantic: a single neuron responds to several unrelated concepts, a phenomenon explained in Lesson 7 by superposition.

Steering and representation engineering

If concepts are represented as directions in activation space, adding or subtracting those directions during a forward pass may change behaviour. Activation steering finds a direction, for example by taking the difference between average activations on honest and dishonest statements, or on positive and negative sentiment, and adds a multiple of it to the model's activations at inference. Studies have shown that such steering can shift sentiment, truthfulness, refusal behaviour and topic. In one well-known study, researchers found that refusal in several chat models was mediated by a single direction, and that removing it disabled refusals, illustrating both the power of these methods for understanding and their potential for misuse. Steering provides causal evidence that a direction matters, because changing it changes behaviour, which correlational probes cannot.

Using interpretability for safety

Interpretability can support safety in several ways: auditing models for hidden objectives or backdoors (Lesson 4), checking whether a model represents information it denies knowing (a sign of dishonesty), detecting evaluation awareness or deceptive reasoning (Lesson 9), explaining failures so that they can be fixed, and providing evidence for safety cases (Lesson 12). In auditing exercises, teams with access to interpretability tools have been able to uncover deliberately implanted hidden objectives in models more effectively than teams limited to behavioural testing. These uses are promising but still developing; current methods explain parts of model behaviour, not all of it.

Concept-based explanations

Attribution methods explain predictions in terms of input features such as pixels or tokens, which are often not the level at which people think. Concept-based methods instead ask how much a human-understandable concept contributed. Testing with concept activation vectors (TCAV), introduced by Kim and colleagues (2018), defines a concept with a set of example inputs (for instance, images of striped textures), learns the direction in activation space that separates them from random examples, and measures how sensitive a class prediction (such as zebra) is to movement along that direction. Concept bottleneck models go further by building the concepts into the architecture: the model first predicts a set of named concepts and then makes its final prediction only from them, so that users can inspect and even correct the intermediate concepts.

Concept methods share the limits of probes: the concept direction is defined by the examples chosen, may capture correlated features rather than the intended concept, and needs validation with random controls and causal tests. They are nonetheless useful for checking whether a model relies on concepts it should not use, such as a medical image classifier relying on hospital markers rather than pathology.

Evaluating explanations

Because plausible explanations are easy to produce, interpretability research increasingly evaluates explanations against criteria such as:

• Faithfulness tests: removing the features an explanation marks as important should change the output more than removing random features.
• Sanity checks: explanations should change when model weights are randomized or labels are shuffled.
• Usefulness to people: explanations should help users predict model behaviour or find bugs in controlled studies.
• Robustness: small, meaning-preserving changes to the input should not produce wildly different explanations.

Choosing a method

No single interpretability method answers every question. Attribution suits explaining individual predictions to users and debugging data problems; probes suit asking whether information is present at a given layer; concept methods suit testing reliance on named concepts; steering suits testing whether a direction causally affects behaviour. A careful investigation often uses several in sequence: a probe to find a candidate representation, a control task to calibrate it, steering to test causality, and evaluation of the resulting explanation for faithfulness. Each method answers a narrower question than its visual output suggests, and conclusions should be stated at that level.