Strategic Chain-of-Thought: Guiding Accurate Reasoning in LLMs through Strategy Elicitation

🌈 Abstract

The article introduces a novel methodology called Strategic Chain-of-Thought (SCoT) to enhance the reasoning capabilities of large language models (LLMs). The key points are:

• SCoT integrates strategic knowledge to guide the generation of high-quality Chain-of-Thought (CoT) paths, addressing the instability and sub-optimal performance of existing CoT methods.
• SCoT employs a two-stage approach within a single prompt: first eliciting an effective problem-solving strategy, which is then used to direct the generation of CoT paths and final answers.
• Experiments across eight challenging reasoning datasets demonstrate significant improvements using SCoT, including a 21.05% increase on the GSM8K dataset and 24.13% on the Tracking_Objects dataset.
• The authors also extend SCoT to a few-shot method with automatically matched demonstrations, yielding even stronger results.

🙋 Q&A

[01] Strategic Knowledge

1. What is strategic knowledge and what are its key principles?

• Strategic knowledge refers to a well-defined method or principle that guides reasoning towards a correct and stable solution.
• It involves using structured processes that logically lead to the desired outcome, thereby enhancing the stability of CoT generation and improving the overall quality of the results.
• Strategic knowledge should adhere to the principles of: 1) Providing a correct and comprehensive problem-solving approach, and 2) Having relatively straightforward problem-solving steps.

2. How does strategic knowledge differ from the approaches used in existing CoT enhancement methods?

• Existing methods like voting-based approaches and retrieval-augmented generation focus on improving the quality of CoT paths, but often come with significant resource demands.
• In contrast, SCoT incorporates strategic knowledge to guide the model in generating high-quality CoT paths, without requiring multiple queries or external knowledge integration.

[02] Strategic Chain-of-Thought (SCoT)

1. What are the key steps of the SCoT method? The SCoT method involves two key steps within a single prompt:

1. Elicitation of Strategic Knowledge: The model identifies and determines one of the most effective and efficient methods for solving the problem, which then serves as the strategic knowledge for the task.
2. Application of Strategic Knowledge: The model subsequently applies the identified strategic knowledge to solve the problem and derive the final answer.

2. How does the few-shot version of SCoT work? The few-shot SCoT method involves two stages:

1. Strategic Knowledge-Based Demonstration Corpus Construction:
   • SCoT is applied to the training set to generate corresponding SCoT answers.
   • The generated answers are compared with ground truth, and only the accurate question-SCoT answer pairs are retained to create a demonstration corpus.
2. Model Inference:
   • The model generates strategic knowledge for the input problem.
   • The generated strategic knowledge is used to search and match the most relevant demonstrations from the corpus.
   • The selected demonstrations are integrated as few-shot examples into the input prompt to guide the model in generating the final prediction.

[03] Experimental Results

1. What are the key findings from the experiments across the eight datasets?

• In zero-shot settings, SCoT outperforms the standard CoT approach in most tasks, with particularly significant improvements on the GSM8K dataset (21.05% increase) and the Tracking_Objects dataset (24.13% increase).
• The few-shot version of SCoT, which combines strategic knowledge and strategy-matched demonstrations, achieves the best results overall.
• SCoT shows substantial gains in commonsense reasoning tasks compared to other methods.

2. How does the effectiveness of SCoT vary across different model sizes?

• Experiments on the Llama2 model series show that SCoT can lead to accuracy improvements across all model sizes.
• However, the performance improvement decreases marginally as the model size increases, as larger models are more likely to generate CoT paths containing strategic knowledge in zero-shot settings.

3. What are the key findings from the ablation study and case study?

• The ablation study shows that adding roles, incorporating workflows, and formatting prompts in Markdown progressively increase the accuracy of SCoT.
• The case study reveals that SCoT tends to favor solving problems using well-defined strategies, such as using inequalities in mathematics, applying the correct formulas in physics, and considering the overall context in multi-hop reasoning tasks, leading to more stable and accurate outputs compared to standard CoT.