Introducing Meta Llama 3: The most capable openly available LLM to date

🌈 Abstract

The article discusses the release of the first two models of the next generation of Llama, Meta Llama 3, which are available for broad use. The key points are:

• Llama 3 features pretrained and instruction-fine-tuned language models with 8B and 70B parameters that can support a broad range of use cases.
• Llama 3 demonstrates state-of-the-art performance on a wide range of industry benchmarks and offers new capabilities, including improved reasoning.
• The goals for Llama 3 were to build the best open models that are on par with the best proprietary models available today, and to address developer feedback to increase the overall helpfulness of Llama 3.
• Llama 3 models are a major leap over Llama 2 and establish a new state-of-the-art for LLM models at the 8B and 70B parameter scale.
• The article discusses the key ingredients in the development of Llama 3, including the model architecture, pretraining data, scaling up pretraining, and instruction fine-tuning.
• Llama 3 will soon be available on all major platforms, and there are plans to release even larger models with new capabilities in the future.

🙋 Q&A

[01] Model Architecture

1. What are the key improvements in the Llama 3 model architecture compared to Llama 2?

• Llama 3 uses a tokenizer with a vocabulary of 128K tokens that encodes language much more efficiently, leading to substantially improved model performance.
• Llama 3 has adopted grouped query attention (GQA) across both the 8B and 70B sizes to improve the inference efficiency of the models.
• Llama 3 was trained on sequences of 8,192 tokens, using a mask to ensure self-attention does not cross document boundaries.

2. How does the Llama 3 architecture align with the design philosophy of the project?

• The article states that in line with the design philosophy of innovating, scaling, and optimizing for simplicity, the Llama 3 project opted for a relatively standard decoder-only transformer architecture.

[02] Training Data

1. What are the key improvements in the Llama 3 training data compared to Llama 2?

• The Llama 3 pretraining dataset is seven times larger than that used for Llama 2, and it includes four times more code.
• Over 5% of the Llama 3 pretraining dataset consists of high-quality non-English data that covers over 30 languages.
• The team developed a series of data-filtering pipelines to ensure Llama 3 is trained on data of the highest quality, including heuristic filters, NSFW filters, semantic deduplication approaches, and text classifiers.

2. How did the team ensure Llama 3 performs well across a variety of use cases?

• The team performed extensive experiments to evaluate the best ways of mixing data from different sources in the final pretraining dataset, enabling them to select a data mix that ensures Llama 3 performs well across use cases including trivia questions, STEM, coding, historical knowledge, etc.

[03] Scaling Up Pretraining

1. What key observations did the team make about scaling behavior during the development of Llama 3?

• The team found that while the Chinchilla-optimal amount of training compute for an 8B parameter model corresponds to ~200B tokens, model performance continues to improve even after the model is trained on two orders of magnitude more data.
• Both the 8B and 70B parameter Llama 3 models continued to improve log-linearly after being trained on up to 15T tokens.

2. What technical improvements did the team make to increase the efficiency of Llama 3 training compared to Llama 2?

• The team combined three types of parallelization: data parallelization, model parallelization, and pipeline parallelization, achieving a compute utilization of over 400 TFLOPS per GPU when trained on 16K GPUs simultaneously.
• The team developed an advanced new training stack that automates error detection, handling, and maintenance, and greatly improved hardware reliability and detection mechanisms for silent data corruption, resulting in an overall effective training time of more than 95%.
• The team developed new scalable storage systems that reduce overheads of checkpointing and rollback.

[04] Instruction Fine-tuning

1. What approaches did the team use for instruction fine-tuning of the Llama 3 models?

• The team's approach to post-training was a combination of supervised fine-tuning (SFT), rejection sampling, proximal policy optimization (PPO), and direct preference optimization (DPO).
• The quality of the prompts used in SFT and the preference rankings used in PPO and DPO had a significant influence on the performance of the aligned models.

2. How did the team's approach to instruction fine-tuning improve the models' capabilities?

• Training on preference rankings via PPO and DPO greatly improved the performance of Llama 3 on reasoning and coding tasks, as it enabled the models to learn how to select the right answer when they knew the reasoning trace.

[05] Responsible Development and Deployment

1. What system-level approach did the team adopt to ensure the responsible development and deployment of Llama 3?

• The team designed Llama 3 models to be maximally helpful while ensuring an industry-leading approach to responsible deployment, by adopting a new, system-level approach that puts the developer in the driver's seat.
• The team's efforts include updating the Llama Guard and CyberSecEval tools, introducing Code Shield for filtering insecure code, and providing comprehensive guidance in the Responsible Use Guide.

2. How did the team ensure the safety of the instruction-fine-tuned Llama 3 models?

• The instruction-fine-tuned Llama 3 models have been extensively red-teamed (tested) for safety through internal and external efforts, leveraging human experts and automation methods to generate adversarial prompts and assess risks of misuse.

[06] Future Plans

1. What are the team's plans for future Llama 3 model releases?

• The article states that the Llama 3 8B and 70B models mark the beginning of what the team plans to release, and that there is a lot more to come, including models over 400B parameters, as well as new capabilities such as multimodality, multilingual support, and longer context windows.
• The team will also publish a detailed research paper once the larger Llama 3 models are fully trained.

2. How does the team plan to continue supporting the open AI ecosystem with Llama 3?

• The team is committed to the continued growth and development of an open AI ecosystem, taking a community-first approach with Llama 3 and making the models available on leading cloud, hosting, and hardware platforms.