How to Prune and Distill Llama-3.1 8B to an NVIDIA Llama-3.1-Minitron 4B Model | NVIDIA Technical Blog

🌈 Abstract

The article discusses the development of small language models (SLMs) as an efficient alternative to large language models (LLMs) for natural language processing tasks. It focuses on the techniques of pruning and knowledge distillation used to obtain smaller models from larger ones, and presents the Llama-3.1-Minitron 4B model as an example.

🙋 Q&A

[01] Pruning and Knowledge Distillation

1. What are the key benefits of pruning and knowledge distillation?

• Pruning and distillation lead to several benefits:
  • Smaller and more efficient models that are cheaper to deploy
  • Retaining much of the predictive power of the original larger model
  • Faster and less resource-intensive to run

2. What are the two main styles of distillation discussed in the article?

• Classical knowledge distillation: Transferring knowledge from a large, complex teacher model to a smaller, simpler student model
• Distillation with structured compression: Combining pruning with classical knowledge distillation as a resource-efficient retraining technique

3. What is the activation-based importance estimation strategy proposed for pruning?

• A purely activation-based importance estimation strategy that simultaneously computes sensitivity information for all the axes (depth, neuron, head, and embedding channel) using a small calibration dataset and only forward propagation passes.
• This is more straightforward and cost-effective compared to strategies relying on gradient information and backward propagation.

4. What are the key structured compression best practices summarized in the article?

• Iterative pruning and importance estimation
• Combining depth, width, attention, and MLP pruning with knowledge distillation-based retraining
• Correcting for distribution shift in the dataset before distillation
• Adopting a width-pruning strategy over depth pruning

[02] Llama-3.1-Minitron 4B Model

1. How was the Llama-3.1-Minitron 4B model obtained from the Llama 3.1 8B model?

• The Llama 3.1 8B model was first fine-tuned on a 94B token dataset to correct for distribution shift.
• 16 layers (50%) were then pruned from the model, with the layers at the beginning and end being the most important.
• Both the embedding and MLP intermediate dimensions were pruned along the width axis.
• The pruned model was then distilled using classical knowledge distillation.

2. How does the performance of the Llama-3.1-Minitron 4B model compare to other similar-sized models?

• The Llama-3.1-Minitron 4B model performs favorably against state-of-the-art open-source models of similar size, including Minitron 4B, Phi-2 2.7B, Gemma2 2.6B, and Qwen2-1.5B, across various benchmarks.
• The width-pruned variant of the Llama-3.1-Minitron 4B model outperforms the depth-pruned variant on most benchmarks.

3. How was the Llama-3.1-Minitron 4B model optimized for inference?

• The Llama 3.1 8B and Llama-3.1-Minitron 4B models were optimized using NVIDIA TensorRT-LLM, an open-source toolkit for optimized LLM inference.
• The Llama-3.1-Minitron-4B-Depth-Base variant achieved an average throughput of ~2.7x compared to the Llama 3.1 8B model, while the Llama-3.1-Minitron-4B-Width-Base variant achieved an average throughput of ~1.8x.
• Deployment in FP8 precision also delivered a performance boost of ~1.3x across all three models compared to BF16.