decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points

🌈 Abstract

The paper proposes a new method called "decoupleQ" for post-training quantization of large language models. The key idea is to decouple the model parameters into an integer part and a floating-point part, and then optimize them alternatively using off-the-shelf optimization methods. This transforms the quantization problem into a constrained optimization problem, rather than relying on traditional heuristic quantization approaches. The method achieves state-of-the-art accuracy, especially at very low bit-widths, and is hardware-friendly due to its uniform quantization. The approach can also be extended to supervised fine-tuning to further improve model accuracy on downstream tasks.

🙋 Q&A

[01] New Insight

1. What is the key insight behind the decoupleQ method? The key insight of decoupleQ is that it abandons the traditional heuristic quantization paradigm and instead models the quantization problem as a constrained optimization problem. By decoupling the model parameters into integer and floating-point parts, it transforms the quantization problem into a mathematical optimization problem that can be solved using off-the-shelf optimization methods, rather than relying on the minutiae of traditional quantization approaches.

2. How does this new approach differ from previous quantization methods? Previous quantization methods remained within the traditional heuristic quantization paradigm, dealing with issues like outliers, sensitive channels, and determining clipping ranges. In contrast, decoupleQ abstracts the essence of the problem and formulates it as a constrained optimization problem, without needing to address these quantization-specific details.

[02] Extreme Low-Bit Quantization

1. What level of quantization does decoupleQ achieve? decoupleQ is able to achieve 2-bit post-training uniform quantization with performance close to fp16/bf16 for industrial applications, such as the ASR model in ByteDance.

2. How does decoupleQ's performance compare to other quantization methods at very low bit-widths? The paper states that existing quantization schemes suffer from significant accuracy degradation at very low bits, while decoupleQ is able to achieve a substantial increase in model accuracy, especially at 2-bit quantization.

[03] Extensibility

1. How can the idea of decoupleQ be extended beyond post-training quantization? The paper mentions that if labeled datasets are available, the idea of decoupleQ can be easily extended to supervised learning to further improve model accuracy or adapt the model to downstream sub-tasks, by freezing the integer part of the model and fine-tuning the floating-point part.

2. What are the potential benefits of this extensibility? The extensibility allows decoupleQ to not only achieve high accuracy in post-training quantization, but also further improve the model's performance on specific downstream tasks by fine-tuning the floating-point part while maintaining the generalization ability of the frozen integer part.