AI Is a Black Box. Anthropic Figured Out a Way to Look Inside

🌈 Abstract

The article discusses the efforts of AI researcher Chris Olah and his team at Anthropic to understand the inner workings of large language models (LLMs) like ChatGPT, Gemini, and Anthropic's own Claude. They are using a technique called "dictionary learning" to identify the combinations of artificial neurons that evoke specific concepts or "features" within the LLM's neural network. This could have significant implications for improving AI safety by allowing them to identify and mitigate potentially dangerous features.

🙋 Q&A

[01] Efforts to Understand Large Language Models

1. What is the key question that has driven Chris Olah's work on artificial neural networks? The key question that has driven Chris Olah's work is "What's going on inside of [artificial neural networks]? We have these systems, we don't know what's going on. It seems crazy."

2. What are some of the challenges with large language models like ChatGPT, Gemini, and Anthropic's Claude? Large language models have dazzled people with their language prowess but also infuriated people with their tendency to make things up. Their potential to solve problems is enticing, but they are also "strangers in our midst" - even the people who build them don't know exactly how they work. Significant effort is required to create guardrails to prevent them from producing bias, misinformation, and even dangerous content like blueprints for deadly weapons.

3. How is Anthropic's approach to understanding large language models different from neuroscience studies on the human brain? Similar to neuroscience studies that interpret MRI scans to identify thoughts in the human brain, Anthropic has plunged into the digital tangle of the neural net of its LLM, Claude, and pinpointed which combinations of its artificial neurons evoke specific concepts or "features." They are essentially trying to reverse engineer the LLM to understand why it comes up with specific outputs.

4. What is the key technique Anthropic is using to identify features in the LLM? Anthropic is using a technique called "dictionary learning" to associate combinations of neurons that, when fired in unison, evoke a specific concept or feature. This involves treating the artificial neurons like letters of the alphabet that can be strung together to have meaning.

[02] Decoding a Full-Size Language Model

1. What were some of the specific features Anthropic's team was able to identify in the Claude language model? The team identified features related to the Golden Gate Bridge, Alcatraz, California governor Gavin Newsom, the Hitchcock movie Vertigo, as well as safety-related features like "getting close to someone for some ulterior motive," "discussion of biological warfare," and "villainous plots to take over the world."

2. How did Anthropic's team try to manipulate the language model's behavior based on the identified features? The team experimented with suppressing or amplifying certain features by "putting a dial" on them. Suppressing dangerous features like unsafe computer code, scam emails, and weapons instructions could make the model safer. But amplifying those features caused the model to become obsessed with them and produce dangerous content like racist screeds and instructions for making weapons.

3. What are some of the limitations of Anthropic's approach to decoding the language model? The techniques used by Anthropic won't necessarily work for decoding other large language models. Additionally, dictionary learning can't identify all the concepts an LLM considers, as it can only identify features that are being explicitly looked for. The picture is therefore bound to be incomplete, though Anthropic believes bigger dictionaries could help mitigate this.

4. How does Anthropic's work compare to efforts by other researchers in this area? Anthropic's work is not the only effort to crack open the black box of LLMs. There are also teams at DeepMind and Northeastern University working on similar problems using different techniques. Olah is encouraged that more people are working on this and that it has gone from being a niche idea to a "decent-sized community" trying to push the field forward.