// category · 8 tools · 6 open source
Interpretability Tools
Libraries and platforms for opening the black box: hooking activations, training sparse autoencoders, attributing outputs and publishing circuit level findings.
Two kinds of interpretability
Tools in this category answer the question of why a model produced an output, but they work at different depths. Attribution libraries such as Captum for PyTorch and SHAP score how much each input feature contributed to a prediction, and they work on almost any model, including classic tabular models. Mechanistic interpretability goes further: it studies a network's weights and activations to find the features it represents and the circuits that connect them. Our mechanistic interpretability explainer introduces the core ideas, from superposition to sparse autoencoders and attribution graphs.
The open source mechanistic toolkit
Versions below were checked on PyPI on October 21, 2026:
- TransformerLens 4.2.0 gives a standard, hook based view of a model. Since version 4.0 its TransformerBridge is the only way to load models.
- nnsight 0.8.1 traces any PyTorch model in place and can run the same code remotely on large open models through NDIF.
- SAELens 6.54.12 trains and loads sparse autoencoders.
- Neuronpedia is a free web platform, open sourced under Apache 2.0 in March 2025, for browsing, searching and steering SAE features and drawing circuit tracing graphs. Its Python client is on PyPI as version 1.2.1.
Our TransformerLens vs nnsight vs Neuronpedia guide shows how they fit into one workflow: find a feature on Neuronpedia, then test it in code.
Where to read the research
Anthropic's Transformer Circuits thread publishes much of the foundational work on features, superposition and circuit tracing, and newer papers appear on arXiv. Companies such as Goodfire build commercial tools on these methods. All of this depends on open weights: you can only look inside a model whose parameters you can download, which is why interpretability research leans on open families such as Gemma, Qwen and GPT 2. If you want a model of your own to study, our guide to training a small LLM from scratch is a practical starting point. Each tool page below links to the project site and its repository or papers.
SHAPshap.readthedocs.ioGame theoretic approach to explaining the output of any machine learning model using Shapley values.InterpretabilityOpen source★ 25.8k
Captumcaptum.aiPyTorch library for model interpretability with integrated gradients, saliency and other attribution methods.InterpretabilityOpen source★ 5.7k
TransformerLensgithub.comLibrary for mechanistic interpretability that exposes and edits the internal activations of GPT style models.InterpretabilityOpen source★ 4k
Goodfiregoodfire.comInterpretability research company building tools to understand and edit the internal workings of neural networks.InterpretabilitySee siteHN 6SAELensgithub.comToolkit for training and analysing sparse autoencoders that decompose model activations into interpretable features.InterpretabilityOpen source★ 1.6k
Neuronpedianeuronpedia.orgFree, open source (Apache 2.0) platform for browsing, searching and steering sparse autoencoder features, with circuit tracing graphs, the Jacobian Lens and a public API.InterpretabilityOpen source★ 1.2k
nnsightnnsight.netLibrary for inspecting and intervening on model internals, with remote execution on large models through NDIF.InterpretabilityOpen source★ 1.1k
Transformer Circuitstransformer-circuits.pubAnthropic's interpretability publication thread, home to influential work on features, superposition and circuit tracing.InterpretabilityFreeHN 1// guides
Guides to interpretability tools
What Is Mechanistic Interpretability Research?
What is mechanistic interpretability research? A plain guide to features, circuits, superposition, sparse autoencoders, key papers, open tools and how to start.
Neuronpedia Guide: TransformerLens vs nnsight
Neuronpedia explained (SAE features, Jacobian Lens, circuit tracer, API) and TransformerLens vs nnsight compared: TransformerLens 4, nnsight 0.8 and NDIF.