Why do small language models underperform? Studying LM Saturation via the Softmax Bottleneck

🌈 Abstract

The article explores the performance saturation phenomenon observed in small language models during training. It finds that this saturation can be explained by a mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution. This mismatch affects the performance of the linear prediction head used in such models through the softmax bottleneck phenomenon. The article measures the effect of the softmax bottleneck in various settings and finds that models based on less than 1000 hidden dimensions tend to adopt degenerate latent representations in late pretraining, leading to reduced evaluation performance.

🙋 Q&A

[01] Why do small language models underperform?

1. Questions related to the content of the section?

• The article finds that small language models can suffer from performance saturation, characterized by a drop in performance at some advanced point in training followed by a plateau.
• This saturation can be explained by a mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution.
• This mismatch affects the performance of the linear prediction head used in such models through the softmax bottleneck phenomenon.
• Models based on less than 1000 hidden dimensions tend to adopt degenerate latent representations in late pretraining, leading to reduced evaluation performance.

[02] Studying LM Saturation via the Softmax Bottleneck

1. Questions related to the content of the section?

• The article measures the cross-entropy of Pythia checkpoints on in-domain data and finds that models up to 410M parameters suffer from the saturation phenomenon.
• The article fits a scaling law on the data and observes that the final checkpoints underperform the extrapolation by 8% on average, while the loss-minimizing checkpoints underperform it by roughly 4%.
• Similar performance saturation is also observed on various evaluation datasets.

2. How does the article characterize the performance saturation of small language models?

• The article characterizes the performance saturation of small language models through evaluation and extrapolation of scaling laws.
• It finds that models up to 410M parameters suffer from the saturation phenomenon, with a drop in performance at some advanced point in training followed by a plateau.

3. What does the article find about the representations of smaller models during saturation?

• The article finds that the representations of smaller models degenerate concurrently with the performance saturation.
• It sheds light on rank saturation, i.e. the explosion of the entropy of singular value distributions of small LM prediction heads.

[03] The Softmax Bottleneck & Language Dimensionality

1. Questions related to the content of the section?

• The article empirically measures a critical value for the rank of the LM head, finding that perplexity starts to noticeably decrease when the rank of the language modeling head is inferior to 1000, regardless of the model size.
• The article estimates the dimensionality inherent to the data itself by training naive 5-gram language models on various datasets and computing the singular value distributions of the resulting matrices.
• The article finds that the estimated rank of the contextual probability distribution is non-negligible with respect to the usual magnitude of hidden dimensions.

2. What does the article find about the inherent dimensionality of natural language?

• The article finds that the rank of the contextual probability distribution is relatively high compared to regular hidden dimension choices in language models.

3. How does the article theoretically analyze the connection between the dimensionality of the ideal language modeling head and performance?

• The article conceptualizes a language modeling head optimized on ideal contextual representations and explores the relationship between its spectral properties and the performance gap induced when training a low-rank head on the same representations.
• It establishes a theoretical link between the performance gap induced by a smaller hidden dimension and the spectral properties of the contextual probability distribution.

[04] Discussion

1. What are some potential ways to address the problem of small language model saturation?

• The article suggests that training shallow small language models with increased hidden dimension at the expense of other hyperparameters may not be a promising direction, as previous work has extensively explored and optimized the hyperparameter choices.
• Another potential way forward could be implementing more expressive softmax alternatives in the context of pretraining small language models on large datasets.

2. What does the article suggest about the nature of the singular components after the collapse observed in small models?

• The article hypothesizes that the resulting dominating components in small models are likely correlated with token frequency, based on previous works linking anisotropy with token frequency and the importance of token frequency in the LM head mechanism.

3. What does the article argue about the implications of its findings?

• The article argues that its work demonstrates that last-layer anisotropy is symptomatic of performance saturation, and is thus likely not a desirable property of language models.
• It also advocates that the work paves the way towards a better understanding of the structure of the contextual probability distribution, which could enhance the interpretation of scaling laws.

</output_format>