SimPO: Simple Preference Optimization with a Reference-Free Reward

🌈 Abstract

The paper proposes SimPO, a simple yet effective offline preference optimization algorithm that outperforms existing approaches across various training setups and benchmarks. The key innovations of SimPO are:

• A length-normalized reward formulation that aligns with the generation likelihood metric, eliminating the need for a reference model.
• Introducing a target reward margin to the Bradley-Terry objective to encourage a larger margin between winning and losing responses.

🙋 Q&A

[01] Direct Preference Optimization (DPO)

1. What is DPO and what are its limitations?

• DPO is a widely used offline preference optimization algorithm that reparameterizes the reward function in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability.
• DPO's limitations are:
  • It requires a reference model during training, incurring additional memory and computational costs.
  • There is a discrepancy between the reward being optimized during training and the generation metric used for inference.

2. How does SimPO address the limitations of DPO?

• SimPO uses a length-normalized reward formulation that directly aligns with the generation likelihood metric, eliminating the need for a reference model.
• SimPO introduces a target reward margin to the Bradley-Terry objective to ensure a larger margin between the winning and losing responses.

[02] SimPO Algorithm

1. What are the key components of the SimPO algorithm?

• Length-normalized reward: SimPO uses the average log probability of a sequence as the implicit reward, which directly aligns with the generation metric.
• Target reward margin: SimPO introduces a target reward margin to the Bradley-Terry objective to encourage a larger margin between the winning and losing responses.

2. How does SimPO's design choices improve performance compared to DPO?

• The length-normalized reward formulation eliminates the need for a reference model, making SimPO more memory and computationally efficient.
• The target reward margin helps to better separate the winning and losing responses, leading to more accurate likelihood ranking of responses.

[03] Experimental Results

1. How does SimPO perform compared to other preference optimization methods?

• SimPO consistently and significantly outperforms existing preference optimization methods, including DPO and its variants, across various state-of-the-art training setups and extensive instruction-following benchmarks.
• SimPO achieves up to a 6.4 point improvement on AlpacaEval 2 and a 7.5 point improvement on Arena-Hard compared to the best baseline.

2. What is the impact of the Instruct setting on model performance?

• The Instruct setting, which uses off-the-shelf instruction-tuned models as the starting point, consistently outperforms the Base setting across all benchmarks.
• This improvement is likely due to the higher quality of the SFT models used for initialization and the generation of more high-quality preference data.