Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

🌈 Abstract

The paper studies how large language models (LLMs) can improve their outputs by using more test-time computation. It analyzes two main mechanisms for scaling test-time compute: (1) searching against dense, process-based verifier reward models, and (2) updating the model's distribution over a response adaptively, given the prompt at test time. The key findings are:

• The effectiveness of different approaches to scaling test-time compute critically varies depending on the difficulty of the prompt.
• Applying a "compute-optimal" scaling strategy, which adaptively allocates test-time compute per prompt, can improve the efficiency of test-time compute scaling by more than 4x compared to a best-of-N baseline.
• On problems where a smaller base model attains somewhat non-trivial success rates, test-time compute can be used to outperform a 14x larger model in a FLOPs-matched evaluation.

🙋 Q&A

[01] Scaling Test-Time Compute via Verifiers

1. What are the key components for using verifiers to scale test-time compute?

• Training process-based reward models (PRMs) that can provide per-step correctness predictions, without requiring human labels
• Exploring different search methods against the PRM, including best-of-N weighted, beam search, and lookahead search
• Finding that the effectiveness of the search methods depends on the difficulty of the prompt, motivating a "compute-optimal" scaling strategy

2. How does the compute-optimal scaling strategy work?

• It selects the best-performing search strategy (e.g. beam search, lookahead search) adaptively for each prompt, based on an estimate of the prompt's difficulty.
• Using oracle difficulty bins, the compute-optimal strategy can outperform a best-of-N baseline by up to 4x, by more effectively allocating the test-time compute budget.
• Even using a model-predicted notion of difficulty, the compute-optimal strategy can still provide substantial improvements over a fixed strategy.

3. How do the different search methods compare in terms of performance?

• Beam search significantly outperforms best-of-N with smaller generation budgets, but the improvements diminish as the budget is scaled up.
• Lookahead search, which uses lookahead rollouts to improve the accuracy of the PRM's value estimation, generally underperforms other methods at the same generation budget.
• The effectiveness of the search methods depends on the difficulty of the prompt, with beam search performing better on harder problems and best-of-N performing better on easier problems.

[02] Refining the Proposal Distribution

1. How do the authors enable LLMs to iteratively refine their own answers?

• They finetune the base LLM to enable it to critique and revise its own outputs in an iterative fashion, using an approach similar to the STaR method.
• This finetuning is necessary because prompting off-the-shelf models is not effective at enabling effective revisions at test time.

2. How do the authors compare sequential revisions vs. parallel sampling?

• They find that on easier problems, sequential revisions outperform parallel sampling, as the model's initial samples are more likely to be on the right track and just need refinement.
• On harder problems, parallel sampling performs better, as it allows the model to explore different high-level problem solving strategies.

3. How do the authors select the best answer from the revision sequence?

• They explore two approaches:
  1. Using a separate ORM verifier trained on the revision model's outputs to select the best answer within each revision trajectory and across trajectories.
  2. Using majority voting across all answers in the revision trajectories.
• They find that majority voting provides smoother scaling behavior compared to the hierarchical ORM-based approach.

[03] Exchanging Pretraining and Test-Time Compute

1. How do the authors define the exchange rate between pretraining and test-time compute?

• They use the common approximation that pretraining FLOPs scale with model parameters and number of pretraining tokens, while inference FLOPs scale with the number of generated tokens.
• They define a ratio R that captures the relative scale of pretraining vs. inference tokens, and analyze the tradeoffs for different values of R.

2. What are the key findings on exchanging pretraining and test-time compute?

• For easier and intermediate questions, and even some harder questions, additional test-time compute can outperform scaling up pretraining, in a FLOPs-matched setting.
• However, on the most challenging questions, scaling up pretraining is more effective than scaling up test-time compute using the current approaches.
• This suggests that while test-time compute scaling can already be preferable to pretraining scaling in some settings, further improvements to test-time compute strategies are needed to make it fully exchangeable with pretraining.

</output_format>