Patch-Level Training for Large Language Models

🌈 Abstract

The paper introduces a novel training approach called "patch-level training" for large language models (LLMs) to improve their training efficiency. The key idea is to reduce the sequence length by compressing multiple tokens into a single "patch", and then train the model to predict the next patch. This allows the model to process the majority of the training data at a significantly reduced computational cost. After the patch-level training, the model continues token-level training on the remaining data to align with the inference mode. Experiments on various model sizes (370M-2.7B parameters) show that this approach can reduce the overall training costs by 50% without compromising model performance.

🙋 Q&A

[01] Patch-Level Training

1. What is the core idea of patch-level training?

• The core idea is to reduce the sequence length by compressing multiple tokens into a single "patch", and then train the model to predict the next patch. This allows the model to process the majority of the training data at a significantly reduced computational cost.

2. How does the patch-level training work?

• The token sequence is first transformed into a patch sequence by compressing every consecutive tokens into a single patch.
• The patch sequence is then fed into the sequence model, and the model is trained to predict all tokens in the next patch.
• The knowledge acquired during patch-level training is subsequently transferred to the token-level model by using the patch-level model parameters to initialize the token-level model, which then continues training on the remaining data.

3. What are the key advantages of patch-level training?

• It can reduce the overall training costs by 50% without compromising model performance.
• It maintains consistency with the subsequent token-level training by setting the patch-level context length to be the same as the token-level context length.
• It avoids introducing unnecessary parameters during token-to-patch compression by representing the patch embedding as the average of its associated token embeddings.

[02] Experiments and Results

1. What are the key findings from the experiments?

• Patch-level training can reduce the overall training costs to 0.5 of the original token-level training, without compromising model performance in terms of perplexity and zero-shot evaluations.
• Patch-level training can also maintain similar instruction-following ability compared to the original token-level models.
• Patch-level training combined with token-level training on the same data can lead to better model regularization and improved performance, especially when the training data is limited.

2. How does the scaling property of patch-level training work?

• As the model size increases, the performance advantage of patch-level training appears to decrease.
• However, as the training data size increases, the performance of patch-level training improves at a faster rate compared to the baseline token-level training.
• This suggests that patch-level training is better suited for scenarios with abundant training data, as more data can facilitate a smoother knowledge transfer from the patch-level to the token-level.

3. What are the effects of the hyperparameters and ?

• The patch size of strikes a favorable trade-off between training efficiency and performance.
• The optimal value of depends on the balance between the benefits of patch-level training and the need for sufficient data to adjust the model to the token-level. Generally, a value around 0.5 seems to work well.

[03] Quantitative Explanation

1. How does patch-level training lead to better learning efficiency?

• In token-level training, only a small proportion of neurons are effectively activated and updated, as the knowledge encapsulated in each token is only associated with a small subset of model parameters.
• By grouping multiple tokens into a patch, the information density processed at each step is increased, leading to higher neuron activation rates and better learning efficiency.

</output_format>