Observational Scaling Laws and the Predictability of Language Model Performance

🌈 Abstract

The article discusses the importance of understanding how language model (LM) performance varies with scale, and proposes an alternative "observational" approach to building scaling laws that leverages predictable log-linear relationships between compute, simple capability measures, and complex downstream metrics. This approach enables accurate predictions on a number of complex LM capabilities, including emergent phenomena, agentic abilities, and the impact of post-training techniques, using only weaker, smaller-scale models.

🙋 Q&A

[01] Understanding Scaling Laws

1. What is the key difference between the standard compute scaling laws and the observational scaling laws proposed in this work? The key difference is that standard compute scaling laws focus on understanding the scaling properties of pretraining, and relate downstream performance to directly controllable quantities like training compute. In contrast, observational scaling laws are interested in scaling laws for downstream, post-training performance, which leads to considering scaling laws across model families and using more directly observable capability measures.

2. How do observational scaling laws generalize the standard compute scaling laws? Observational scaling laws hypothesize the existence of a low-dimensional capability measure for LMs that relates compute to more complex LM capabilities, and can be extracted from observable standard LM benchmarks. This allows observational scaling laws to be applied across multiple model families, whereas standard compute scaling laws are limited to a single model family.

3. What are the key advantages of observational scaling laws compared to standard compute scaling laws? The key advantages are:

• Lower cost: Observational scaling incurs no training cost, leveraging existing models.
• Higher resolution: Observational scaling uses a larger number of models spanning a much larger compute range.
• Broader coverage: Observational scaling can combine model families with heterogeneous scaling properties and capabilities.

[02] Validating Observational Scaling Laws

1. How do the authors validate the predictive power of observational scaling laws? The authors design experiments with systematic holdout sets, where they fit the scaling laws on weaker models and evaluate the extrapolated predictions on stronger held-out models. They also preregister their fitted scaling laws and commit to updating their prediction accuracy on future models.

2. What are some of the complex phenomena that observational scaling laws are able to accurately predict? The authors show that observational scaling laws can accurately predict:

• Emergent capabilities that were previously thought to be discontinuous
• The performance of agentic capabilities of LMs, as measured by AgentBench and AgentBoard
• The impact of post-training techniques like Chain-of-Thought and Self-Consistency as model scale increases

3. How do the authors select a small subset of models that maintain high prediction accuracy for practical scaling analyses? The authors use an optimal experimental design approach to identify the optimal subset of models, subject to a budget constraint on the number of models. This allows them to provide concrete and practical prescriptions for researchers and practitioners to perform similar scaling analyses.