What is mechanistic interpretability research? It is the study of how a neural network works on the inside: researchers look at a trained model's weights and activations and try to work out the algorithms it has learned, the way an engineer reverse engineers a compiled program. Instead of only asking "what did the model output?", they ask "which internal parts produced this output, and how?" The field's basic objects are features (the concepts a model represents) and circuits (the connections between features that carry out a computation), and its main tools are open libraries you can run on open weight models today.
What does mechanistic interpretability mean?
The word "mechanistic" is the key. Plenty of interpretability work explains a model from the outside. Methods such as saliency maps, SHAP and LIME estimate which parts of an input mattered for a single prediction. Mechanistic work goes further and tries to explain the computation itself.
The 2025 review Open Problems in Mechanistic Interpretability, written by researchers from Apollo Research, Anthropic, Google DeepMind, MIT and other groups, says the field "aims to understand the computational mechanisms underlying neural networks' capabilities." It sets mechanistic work apart from older threads that build simple models that are interpretable by design, or that explain one decision at a time. Mechanistic work asks how a model solves a whole class of problems.
Researchers do not all define it the same way. Neel Nanda, who runs a mechanistic interpretability team at Google DeepMind, offers a broad working definition in his guide to becoming a mech interp researcher: mechanistic interpretability is any approach to understanding a model that uses its internals, meaning its weights and activations.
| Approach | What it looks at | Typical question |
|---|---|---|
| Black box evaluation | Inputs and outputs only | Does the model get this right? |
| Attribution methods (saliency, SHAP, LIME) | How sensitive the output is to each part of the input | Which words or pixels mattered here? |
| Concept probes | Whether a concept can be read out of hidden activations | Does the model encode "this is false"? |
| Mechanistic interpretability | Weights, activations and the paths between them | What algorithm produces this behaviour? |
Features, circuits and superposition
Most of the field's vocabulary comes from one essay. In Zoom In: An Introduction to Circuits (Distill, March 2020), Chris Olah and colleagues, then at OpenAI, made three deliberately speculative claims about vision networks:
- Features are the basic unit of a neural network, and they correspond to directions in activation space (a direction can be one neuron or a mix of many).
- Circuits form when features are connected by weights, and they can be studied rigorously.
- Universality: similar features and circuits form across different models and tasks.
The essay's examples came from the InceptionV1 image model. It traced curve detectors, found that they are built from earlier and simpler curve detectors, and showed that the authors could rebuild them by setting weights by hand. It also documented a catch: some neurons are polysemantic. One neuron responded to cat faces, the fronts of cars and cat legs, which are unrelated things.
The authors' explanation was superposition: a network can pack more features than it has neurons by spreading each feature across many neurons. That works because high dimensional spaces have room for far more nearly perpendicular directions than true axes. Superposition is good for the model and bad for the researcher, because you can no longer read meaning off individual neurons. Much of the field since 2022 has been about getting around this problem.
What is a circuit in mechanistic interpretability?
A circuit is a small piece of a network, made of a few features and the weighted connections between them, that performs a recognisable computation. The Zoom In glossary defines it as a subgraph of the network. In language models, two results became textbook cases.
Induction heads. Anthropic's In context Learning and Induction Heads (2022) describes attention heads that complete patterns of the form [A][B] … [A] → [B]: if a sequence appeared earlier in the text, the model predicts it will repeat. The paper reports that these heads appear at the same point in training as a sudden jump in in context learning ability. It also notes that the evidence is strong and causal for small attention only models, and only correlational for larger ones.
Indirect object identification. In Interpretability in the Wild (2022), researchers explained how GPT-2 small completes sentences like "When Mary and John went to the store, John gave a drink to …" with "Mary". Their circuit spans 26 attention heads grouped into 7 classes. They tested it for faithfulness, completeness and minimality, and those three criteria are now a standard way to judge a circuit.
Circuits are tested by causal intervention. Activation patching, for example, copies an internal activation from one prompt into another prompt's forward pass and measures how the answer changes. If the circuit story is right, editing the parts it names should change the output in the way it predicts.
Sparse autoencoders: pulling features out of superposition
To deal with polysemantic neurons, researchers began using sparse autoencoders (SAEs). An SAE is a small extra network trained to rebuild a model's activations from a much larger set of features, of which only a few are active at any moment. Each learned feature tends to be cleaner than any single neuron.
- In September 2023, Cunningham and colleagues reported that SAE features in language models are more interpretable and monosemantic than directions found by alternative methods.
- In October 2023, Anthropic's Towards Monosemanticity applied the method to a one layer transformer, and in May 2024 Scaling Monosemanticity applied it to Claude 3 Sonnet. Both are listed on Anthropic's Transformer Circuits site.
- In his essay The Urgency of Interpretability, Anthropic CEO Dario Amodei writes that the team found over 30 million features in Claude 3 Sonnet. Turning up a "Golden Gate Bridge" feature produced Golden Gate Claude, a version that brought up the bridge even in unrelated conversations.
- In August 2024, Google DeepMind released Gemma Scope, an open suite of SAEs trained on all layers of Gemma 2 2B and 9B and selected layers of 27B, so researchers outside big labs could use them.
SAEs have well documented limits. The Open Problems review notes that when a 16 million feature SAE's reconstruction was put back into GPT-4, the model's loss matched a model trained with about 10% of GPT-4's pretraining compute. In other words, the SAE misses part of what the model computes. Nanda's guide argues that the wave of small, incremental SAE papers has run its course, while SAEs remain useful as tools.
From features to attribution graphs
In March 2025, Anthropic published Circuit Tracing and On the Biology of a Large Language Model, both on Transformer Circuits. They study Claude 3.5 Haiku with attribution graphs. According to the Neuronpedia circuit tracing docs, the method swaps the model's MLP layers for transcoders, whose features are easier to read. It then computes how much each feature, token and error term contributes to the next.
The standard example is the prompt "The capital of the state containing Dallas is". The graph shows a path of roughly Dallas → Texas → capital → Austin, so the model takes an internal step through "Texas" instead of jumping straight to the answer. Amodei's essay adds that the same tracing showed models planning rhymes ahead when writing poetry and sharing concepts across languages.
Anthropic then open sourced its circuit tracing tools as the circuit-tracer library, with a hosted front end on Neuronpedia (our TransformerLens vs nnsight vs Neuronpedia guide explains how they fit together). Neuronpedia's docs say you can generate new graphs there for Gemma 2 2B, Gemma 3 4B Instruct, Qwen3 4B and Qwen3 1.7B, all of them open weight models.
Mechanistic interpretability at Anthropic and beyond
People often search "mechanistic interpretability Anthropic" because Anthropic publishes the field's best known papers. The Transformer Circuits index shows a steady stream in 2026: in April, work finding representations of emotion concepts in Claude Sonnet 4.5 that causally affect its outputs; in May, natural language autoencoders that train Claude to translate its internal state into words; and in July, a paper on a small set of representations the model can report on and control.
Anthropic is not the only group. The Open Problems review lists authors from Google DeepMind, EleutherAI, MIT, Northeastern University, Goodfire, Decode Research and others. Goodfire, a San Francisco start up, released a tool called Silico in April 2026. MIT Technology Review reports that it lets researchers inspect neurons in open models and adjust parameters during training. The same magazine named mechanistic interpretability one of its 10 Breakthrough Technologies of 2026, while noting that the field is split on how far these techniques can go.
Mechanistic interpretability for AI safety
Safety is the main reason many researchers work in this area. The Open Problems review lists goals such as monitoring models for signs of dangerous cognition, editing internal mechanisms, and predicting behaviour in situations nobody has tested. Its argument is that if a model can behave well during an evaluation while reasoning badly inside, only methods that look inside can catch it.
Amodei's essay describes the long term aim as an "MRI for AI" and states a goal of getting to "interpretability can reliably detect most model problems" by 2027. Nanda's guide is more cautious. He writes that completely reverse engineering a model now looks very unlikely to him, and he favours practical uses, such as cheap linear probes for monitoring. Both views are common in the field, and both rely on the same methods and open tools described here.
Mechanistic interpretability at CVPR and in vision
The field started with vision models, and it still has a home at computer vision conferences. CVPR 2025 hosted a Mechanistic Interpretability for Vision (MIV) workshop. Its follow up at CVPR 2026 in Denver, the How Do Vision Models Work? (HOW) workshop, widened the scope to how vision models learn, represent and reason. Its papers are published in the CVF open access proceedings.
How to start doing mechanistic interpretability research
Mechanistic interpretability needs far less compute than training frontier models, which is one reason it suits independent researchers and anyone already running models locally. Nanda's guide recommends learning the basics for no more than a month and then moving straight to small projects of 1 to 5 days each.
- Get the foundations. Linear algebra matters most. Then code a small transformer such as GPT-2 in PyTorch; the train a small LLM from scratch guide is a good companion.
- Learn the core techniques on a small model: activation patching, linear probes, max activating dataset examples, using a pretrained SAE, steering vectors and the logit lens.
- Pick a library. TransformerLens is the teaching standard. nnsight wraps any PyTorch model and can run on large models remotely. Our TransformerLens vs nnsight comparison explains how to choose. SAELens handles sparse autoencoders.
- Explore without code. Browse features and make attribution graphs on Neuronpedia.
- Read selectively. Start with Zoom In, the Open Problems review and the Transformer Circuits archive, and find newer papers on arXiv, where alphaXiv adds line by line discussion, or with their peer reviews on OpenReview.
- Use open weights. You can only look inside models whose weights you have. Our explainer on open weight vs open source models covers which licences allow this, and the VRAM guide tells you what fits on your GPU.
FAQ
Is mechanistic interpretability the same as explainable AI?
They overlap but are not the same. Explainable AI often means attribution methods such as SHAP or saliency maps, which show which inputs mattered for one prediction. Mechanistic interpretability tries to explain the internal computation, meaning the features and circuits, that produces a whole class of behaviour.
What is superposition in mechanistic interpretability?
Superposition means a model stores more features than it has neurons by spreading each feature across many neurons. It explains why single neurons often respond to unrelated things, and it is the main reason researchers use sparse autoencoders to separate features.
Who started mechanistic interpretability?
Amodei's essay credits Chris Olah with one of the first systematic attempts to open the black box, a research programme that "has come to be known as mechanistic interpretability". Olah's 2020 Zoom In essay set out the field's core ideas.
Can I do mechanistic interpretability research on my own computer?
Yes, on small open models. Nanda's guide notes that the field needs modest compute, that Google Colab works to begin with, and that newer MacBook Pros or gaming GPUs may run models locally. nnsight's remote service can reach much larger models.
Does mechanistic interpretability work on closed models like ChatGPT?
Not from the outside. You need the weights and activations, so outside researchers work on open weight models such as Gemma, Qwen and Llama. Labs like Anthropic study their own closed models internally and publish the results.
Is mechanistic interpretability a good research field to join?
It is growing and still has many open questions; the Open Problems review lists many unsolved problems in methods and applications. It also has free learning material and open tools, so you can build a public portfolio before applying to labs, fellowships or PhD programmes.
