github.com · Interpretability
Toolkit for training and analysing sparse autoencoders that decompose model activations into interpretable features.
SAELens is one of 8 interpretability tools on Axiomi. See all interpretability tools or how we check facts.
// guides
What is mechanistic interpretability research? A plain guide to features, circuits, superposition, sparse autoencoders, key papers, open tools and how to start.
Neuronpedia explained (SAE features, Jacobian Lens, circuit tracer, API) and TransformerLens vs nnsight compared: TransformerLens 4, nnsight 0.8 and NDIF.
// related
Goodfiregoodfire.comInterpretability research company building tools to understand and edit the internal workings of neural networks.InterpretabilitySee siteHN 6
Neuronpedianeuronpedia.orgFree, open source (Apache 2.0) platform for browsing, searching and steering sparse autoencoder features, with circuit tracing graphs, the Jacobian Lens and a public API.InterpretabilityOpen source★ 1.2k
nnsightnnsight.netLibrary for inspecting and intervening on model internals, with remote execution on large models through NDIF.InterpretabilityOpen source★ 1.1k
SHAPshap.readthedocs.ioGame theoretic approach to explaining the output of any machine learning model using Shapley values.InterpretabilityOpen source★ 25.8k
Captumcaptum.aiPyTorch library for model interpretability with integrated gradients, saliency and other attribution methods.InterpretabilityOpen source★ 5.7kTransformerLensgithub.comLibrary for mechanistic interpretability that exposes and edits the internal activations of GPT style models.InterpretabilityOpen source★ 4k