Qwen2 Technical Report

🌈 Abstract

This report introduces the Qwen2 series, the latest addition to the Qwen family of large language models and large multimodal models. The Qwen2 series includes a comprehensive suite of foundational and instruction-tuned language models, ranging from 0.5 to 72 billion parameters, featuring dense models and a Mixture-of-Experts model. Qwen2 surpasses most prior open-weight models, including its predecessor Qwen1.5, and exhibits competitive performance relative to proprietary models across diverse benchmarks on language understanding, generation, multilingual proficiency, coding, mathematics, and reasoning.

🙋 Q&A

[01] Tokenizer & Model

1. What are the key features of the Qwen2 tokenizer?

• The Qwen2 series employs the same tokenizer based on byte-level byte-pair encoding as the previous Qwen models.
• This tokenizer exhibits high encoding efficiency, with a better compression rate compared to alternatives, facilitating the multilingual capabilities of Qwen2.
• The common vocabulary consists of 151,643 regular tokens and 3 control tokens.

2. What are the architectural differences between the Qwen2 dense models and the previous Qwen models?

• Qwen2 dense models incorporate Grouped Query Attention and Dual Chunk Attention with YARN to expand the context window and improve long-context performance.
• The Qwen2 Mixture-of-Experts (MoE) model uses fine-grained experts, creating smaller-scale experts and activating a greater number of experts simultaneously to enhance performance and adaptability.
• The MoE model also employs shared and routing-specific experts to facilitate more adaptable and efficient expert utilization.

3. What are the key configurations of the Qwen2 model series?

• The Qwen2 series consists of 5 model sizes: Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B.
• Qwen2 models demonstrate a substantially lower Key-Value (KV) size per token compared to Qwen1.5 models, reducing the memory footprint, particularly advantageous in long-context inference tasks.

[02] Pre-training

1. How has the pre-training data for Qwen2 been improved compared to previous Qwen models?

• The pre-training data for Qwen2 has been expanded from 3 trillion tokens in Qwen1.5 to 7 trillion tokens, with a focus on enhancing the quality, scale, and diversity of the data, particularly in the areas of code, mathematics, and multilingual content.
• The filtering algorithm has been refined with additional heuristic and model-based methods, including the use of Qwen models to filter out low-quality data and synthesize high-quality pre-training data.

2. How has the Qwen2 models been trained to handle long-context scenarios?

• During the concluding phase of pre-training, the context length was expanded from 4,096 tokens to 32,768 tokens, accompanied by a significant increase in the volume of high-quality, lengthy data.
• The base frequency of RoPE was modified from 10,000 to 1,000,000 to optimize performance in long-context scenarios.
• The Qwen2 models incorporate the YARN mechanism and Dual Chunk Attention to enable processing sequences of up to 131,072 tokens while maintaining high performance.

[03] Post-training

1. What are the key components of the post-training process for Qwen2?

• The post-training process involves Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to enhance the models' proficiency across a broad spectrum of domains, including coding, mathematics, logical reasoning, instruction following, and multilingual comprehension.
• The post-training data consists of demonstration data and preference data, which are obtained through a combination of collaborative data annotation and automated data synthesis.

2. How does the post-training process ensure the safety and responsibility of the Qwen2 models?

• The post-training process includes multilingual safety evaluations that test the models' ability to reject responses related to illegal behaviors, fraud, pornography, and privacy violations.
• Compared to the proprietary model GPT-4 and the open-weight model Mixtral-8x22B-Instruct, Qwen2-72B-Instruct demonstrates better performance in terms of safety and responsibility, although there is still room for improvement, especially in the area of pornography.

[04] Evaluation

1. How do the Qwen2 base language models perform compared to other state-of-the-art models?

• Qwen2-72B outperforms Llama-3-70B and Qwen1.5 models across various benchmarks, including language understanding, coding, mathematics, and Chinese language tasks.
• The Qwen2-57B-A14B MoE model also demonstrates competitive performance compared to dense models of similar parameter counts, such as Yi-1.5-34B and Qwen1.5-32B.
• The smaller Qwen2 models, such as Qwen2-0.5B and Qwen2-1.5B, also exhibit significant improvements over their Qwen1.5 counterparts.

2. How do the Qwen2 instruction-tuned models perform compared to other state-of-the-art models?

• Qwen2-72B-Instruct outperforms Mixtral-8x22B-Instruct, Llama-3-70B-Instruct, and Qwen1.5-72B-Chat across various benchmarks, including language understanding, coding, mathematics, and human preference alignment tasks.
• Qwen2-57B-A14B-Instruct also demonstrates superior performance compared to Mixtral-8x7B-Instruct and the dense SOTA models like Yi-1.5-34B-Chat and Qwen1.5-32B-Chat.
• The Qwen2-7B-Instruct model exhibits competitive performance with Llama-3-8B-Instruct, particularly in coding tasks, but falls behind in instruction-following abilities, which the authors plan to address in future iterations.

3. How do the Qwen2 models perform in evaluations of long-context capabilities?

• Qwen2-72B-Instruct and Qwen2-7B-Instruct demonstrate exceptional accuracy in retrieving information from contexts up to 128,000 tokens, outperforming the baseline models.
• The integration of YARN and Dual Chunk Attention significantly improves the long-context capabilities of the Qwen2 models, enabling them to handle contexts up to 256,000 tokens with high performance.

4. How do the Qwen2 models perform in the evaluation of multilingual capabilities?

• Qwen2-72B-Instruct exhibits competitive multilingual performance, outperforming GPT-3.5-Turbo and matching the performance of GPT-4-Turbo and Claude-3-Opus across a range of languages, including Arabic, French, Indonesian, Japanese, Korean, Portuguese, Russian, Spanish, Thai, and Vietnamese.

</output_format>