![magic star](/img/share-magic.png?1753566981)
What happened to BERT & T5? On Transformer Encoders, PrefixLM and Denoising Objectives — Yi Tay
🌈 Abstract
The article discusses the evolution of language model architectures in the era of large language models (LLMs), covering the three main paradigms: encoder-only models (e.g., BERT), encoder-decoder models (e.g., T5), and decoder-only models (e.g., GPT series). It explores the relationships between these architectures, the role of denoising objectives, and the gradual phasing out of BERT-like models in favor of more flexible and unified autoregressive models.
🙋 Q&A
[01] Encoder-only, Encoder-Decoder, and Decoder-only Models
1. What are the key differences between encoder-only, encoder-decoder, and decoder-only models?
- Encoder-only models like BERT use a denoising objective where the model learns to predict masked tokens in-place.
- Encoder-decoder models like T5 are also autoregressive, with the decoder being a causal decoder that can leverage the encoder's representations via cross-attention.
- Decoder-only models like the GPT series are autoregressive language models that predict the next token based on the previous ones.
2. How are encoder-decoder and decoder-only models related?
- Encoder-decoder models and decoder-only models are not fundamentally different - they are both autoregressive models, with the key difference being the presence of an encoder in encoder-decoder models.
- A variant of encoder-decoder models is the Prefix Language Model (PrefixLM) architecture, which is a decoder-only model that can leverage bidirectional attention without the need for a separate encoder.
3. Why did BERT-like encoder-only models become less prominent compared to encoder-decoder and decoder-only models?
- The shift towards more unified, multi-task models made BERT-like models less desirable, as encoder-decoder and decoder-only models could more easily express multiple tasks without the need for task-specific classification heads.
- Encoder-decoder and decoder-only models also retained the bidirectional attention benefits of BERT while offering more flexibility and efficiency.
[02] Denoising Objectives
1. What is the denoising objective, and how does it differ from regular language modeling?
- The denoising objective, also known as "span corruption" or "infilling", involves masking and predicting a subset of tokens in the input sequence, rather than predicting the next token in a causal language modeling objective.
- In denoising objectives, the masked tokens are "moved to the back" of the sequence for the model to predict, whereas in regular language modeling, the model predicts the next token in a left-to-right fashion.
2. How effective are denoising objectives compared to regular language modeling?
- Denoising objectives are seen as a complementary objective to regular language modeling, as they can learn useful representations in a more sample-efficient way for certain tasks.
- However, denoising objectives have the drawback of "less loss exposure" since only a small portion of tokens are masked and predicted, compared to the full sequence in regular language modeling.
- The author suggests that denoising objectives should be used in conjunction with regular language modeling, rather than as a standalone pretraining objective.
[03] Bidirectional Attention and Model Scaling
1. What is the role of bidirectional attention in language models?
- Bidirectional attention, as seen in BERT-style models, is an inductive bias that can be beneficial at smaller scales, as it allows the model to leverage both past and future context.
- However, the author suggests that the importance of bidirectional attention may diminish at larger scales, and that its impact can vary depending on the task or modality.
2. How do encoder-decoder architectures compare to decoder-only models in terms of advantages and drawbacks?
- Encoder-decoder models can be more flexible in the encoder side, as they are not restricted by the causal mask like decoder-only models. This allows for more aggressive pooling or linear attention in the encoder.
- However, encoder-decoder models have the drawback of fixed input and output budgets, whereas decoder-only models like PrefixLM can more efficiently concatenate input and target sequences.
[04] Recap and Conclusions
1. What are the key takeaways from the article?
- Encoder-decoder and decoder-only models are both autoregressive models with subtle differences in inductive biases and implementation-level details.
- Denoising objectives are mostly used as complementary objectives to regular causal language modeling, rather than as standalone pretraining objectives.
- Bidirectional attention can be beneficial at smaller scales, but its importance may diminish at larger scales, depending on the task and application.
- BERT-like encoder-only models have become less prominent, as more flexible and unified autoregressive models like T5 and PrefixLM have emerged, allowing for better multi-task capabilities.