Understanding Anthropic’s Golden Gate Claude

🌈 Abstract

The article discusses the release of "Golden Gate Claude", a manipulated version of Anthropic's Claude 3 Sonnet model that had an obsession with the Golden Gate Bridge. It explores the process the authors went through to create Golden Gate Claude and, more generally, to identify interpretable features that can be used to explain and manipulate large language models (LLMs). The article also discusses the potential of using these techniques to make LLMs more interpretable and safer.

🙋 Q&A

[01] The Process of Creating Golden Gate Claude

1. What was the motivation behind creating Golden Gate Claude? The researchers at Anthropic wanted to explore techniques for tackling two of the biggest challenges posed by large language models (LLMs): interpretability and safety. The creation of Golden Gate Claude was a way to demonstrate these techniques.

2. How did the researchers create Golden Gate Claude? The researchers used a sparse autoencoder (SAE) to identify interpretable features in the middle layer of the Claude 3 Sonnet model. They then used a technique called "feature clamping" to manipulate the model's behavior, making it obsessed with the Golden Gate Bridge.

3. What were some of the key interpretable features identified by the researchers? In addition to the Golden Gate Bridge feature, the researchers identified other interpretable features such as brain sciences, monuments and popular tourist attractions, transit infrastructure, code errors, inner conflict, and emotional expressions.

4. How did the researchers demonstrate the interpretability of the identified features? The researchers showed that as a token moves further from the concept represented by a feature, the strength of the feature's activation decreases. They also plotted the nearest neighbors of the features using cosine similarity, demonstrating that features representing similar concepts were closer together.

[02] Implications for Interpretability and Safety of LLMs

1. How can the techniques used to create Golden Gate Claude help improve the interpretability of LLMs? The ability to identify interpretable features in the inner workings of LLMs can help open up these models for use in applications with greater requirements for explainability. This is a significant finding for the development of LLMs.

2. How can these techniques be used to improve the safety of LLMs? The researchers explored "feature steering", where the identified features can be used to influence the behavior of the model during inference. This could be used to steer the model away from unsafe or biased behaviors, making the models fundamentally safer than current LLMs.

3. What are the advantages of making LLMs safer through feature manipulation, compared to other techniques? Manipulating the model's layer activations during inference can reduce the need for manual safety processes added on top of the model, and also reduce the chance that users can find ways around the safety measures.