Inside One of the Most Important Papers of the Year: Anthropic’s Dictionary Learning is a…

🌈 Abstract

The article discusses the importance of interpretability in large language models (LLMs) and Anthropic's work on using dictionary learning to extract interpretable features from their Claude Sonnet model.

🙋 Q&A

[01] Interpretability in LLMs

1. What are the concerns around the lack of interpretability in LLMs?

• LLMs are often seen as opaque systems, where the reasoning behind their responses remains hidden, which complicates their trustworthiness and raises concerns about their potential to produce harmful, biased, or untruthful outputs.
• Examining the internal state of the model, which is a collection of neuron activations, does not necessarily clarify things, as the concepts the model comprehends and utilizes cannot be directly discerned by examining the neurons.

2. What is Anthropic's approach to improving interpretability?

• Anthropic published work on matching neuron activation patterns, termed "features", to concepts understandable by humans using "dictionary learning" from classical machine learning.
• This allows the model's internal state to be represented by a few active features instead of many active neurons, making the model more interpretable.

[02] Anthropic's Dictionary Learning Approach

1. What is the core technique used by Anthropic?

• Anthropic used sparse autoencoders (SAEs) to decompose model activations (specifically in the Claude 3 Sonnet model) into more interpretable "features".
• The SAE comprises an encoder layer that maps activity to a higher-dimensional layer of features, and a decoder layer that attempts to reconstruct the model activations from the feature activations.
• The training objective minimizes reconstruction error and an L1 regularization penalty on feature activations, promoting sparsity.

2. How do Anthropic evaluate the quality of the extracted features?

• Anthropic uses the loss function during training, which is a combination of reconstruction mean-squared error (MSE) and an L1 penalty on feature activations, as a proxy for feature quality.
• They also perform automated interpretability experiments to evaluate a larger number of features and compare them to neurons.

3. How do Anthropic demonstrate the relevance of the extracted features?

• Anthropic experiments with "feature steering", where they clamp specific features to high or low values during the forward pass, to modify the model's outputs in specific, interpretable ways.
• This provides evidence that the feature interpretations align with the model's use of the features.
• Anthropic also explores the local neighborhoods of several features, finding related features or contexts within these neighborhoods.

4. What are the advantages of Anthropic's dictionary learning approach compared to other methods?

• Dictionary learning offers unique advantages over other methods, such as using linear probes and activation steering, by leveraging the insights gained from decomposing the model's activations into interpretable features.

5. What is the broader significance of Anthropic's work?

• Anthropic's work suggests that the interpretability of LLMs might be a scaling problem that can be solved, opening the door to a future where these systems are both understandable and trustworthy.