Distilling System 2 into System 1

🌈 Abstract

The article discusses distilling System 2 reasoning techniques, which involve generating intermediate thought processes, into a more efficient System 1 model that directly generates the final output. The authors explore several System 2 methods, including Rephrase and Respond (RaR), System 2 Attention (S2A), and Branch-Solve-Merge (BSM), and investigate how to distill their benefits into a single System 1 model. They find that this distillation is often successful, leading to improved performance compared to the original System 1 model while requiring less computational cost. However, they also note that not all System 2 techniques, such as Chain-of-Thought (CoT) for complex reasoning tasks, can be easily distilled.

🙋 Q&A

[01] Distilling System 2 into System 1

1. What is the key difference between System 1 and System 2 models?

• System 1 models directly generate the final output without any intermediate steps, while System 2 models generate intermediate thought processes before producing the final output.

2. What are the main goals of System 2 distillation?

• The main goals of System 2 distillation are to:
  • Distill the reasoning and benefits of System 2 methods into a more efficient System 1 model
  • Maintain the performance improvements of System 2 while reducing the computational cost and inference time

3. How does the authors' approach for System 2 distillation work?

• The authors use a self-supervised approach where they:
  1. Apply various System 2 methods (RaR, S2A, BSM) to unlabeled data to generate intermediate outputs and final responses
  2. Use consistency filters (e.g., self-consistency, input perturbation) to select high-quality distillation targets
  3. Fine-tune the base System 1 model to match the distilled System 2 outputs, without generating the intermediate steps

4. What are the key findings from the experiments on distilling different System 2 methods?

• The authors successfully distilled RaR, S2A, and BSM into System 1 models, often achieving performance improvements over the original System 1 baseline while reducing computational cost.
• However, they found that distilling the Chain-of-Thought (CoT) method for complex reasoning tasks was not as successful, highlighting the limitations of their distillation approach.

[02] Rephrase and Respond (RaR) Distillation

1. How does the RaR method work, and what tasks did the authors evaluate it on?

• RaR is a two-stage System 2 method that first prompts the model to rephrase the original question and then generate a response based on the rephrased question.
• The authors evaluated RaR on the last letter concatenation task and the coin flip reasoning task.

2. What were the key results from distilling RaR into a System 1 model?

• For the last letter concatenation task, distilling 2-step RaR into a System 1 model achieved 98% accuracy, significantly outperforming the original System 1 baseline (30% accuracy).
• For the coin flip reasoning task, distilling 2-step RaR into a System 1 model achieved 75.69% accuracy, comparable to the 2-step RaR System 2 model (77.2%) but with much lower computational cost.

3. What was the role of the self-consistency filtering in the distillation process?

• The authors found that the self-consistency filtering step was critical for the quality of the distillation data, as distilling without this filtering step led to significantly lower performance.

[03] System 2 Attention (S2A) Distillation

1. What is the key idea behind the S2A method?

• S2A is a two-stage System 2 method that first rewrites the input to remove biased or irrelevant information, and then generates the final response based on the rewritten context.

2. How did the authors evaluate the distillation of S2A?

• The authors evaluated S2A distillation on the SycophancyEval task, which contains biased information in the input that can hurt LLM performance.

3. What were the key results from distilling S2A into a System 1 model?

• The distilled S2A System 1 model outperformed both the original System 1 baseline and the System 2 S2A model in terms of agreement with human judgments, while using significantly fewer output tokens.

[04] Branch-Solve-Merge (BSM) Distillation

1. What is the key idea behind the BSM method?

• BSM is a System 2 method that breaks down a task into several parallel sub-tasks, evaluates each sub-task independently, and then merges the results to produce the final output.

2. How did the authors evaluate the distillation of BSM?

• The authors evaluated BSM distillation on the Open Assistant Dataset v2 (OASST2) and the MT-bench benchmark, which evaluate LLMs as judges of other LLM responses.

3. What were the key results from distilling BSM into a System 1 model?

• The distilled BSM System 1 model outperformed both the original System 1 baseline and the System 2 BSM model in terms of agreement with human judgments, while using significantly fewer output tokens.

[05] Limitations and Future Directions

1. What were the key limitations of the authors' System 2 distillation approach?

• The authors found that not all System 2 methods, such as Chain-of-Thought (CoT) for complex reasoning tasks, could be effectively distilled into a System 1 model using their approach.
• The performance of the distillation process also relied on the quality of the self-supervised filtering techniques used to select high-quality distillation targets.

2. What are some potential future research directions suggested by the authors?

• The authors suggest that exploring System 2 distillation in a continuous training loop, similar to how humans learn to transfer skills from deliberate to automatic processing, could be a fruitful research direction.
• They also note that further work is needed to understand the specific circumstances in which System 2 distillation is effective, and when it may not be suitable.