Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, Lu Hou
2025-04-08
Summary
This paper talks about how making AI models smaller and faster (quantization) affects their ability to solve math problems and think logically, showing where it helps and where it causes mistakes.
What's the problem?
Shrinking AI models to run faster often makes them worse at solving complex problems, especially math and science questions, which could lead to wrong answers in important tasks.
What's the solution?
Researchers tested different levels of 'shrinking' on AI models and found that keeping some parts less compressed (like using 8-bit or 4-bit precision) works better, and adding more thinking steps helps models stay accurate.
Why it matters?
This helps developers create faster AI tools for phones and small devices without sacrificing accuracy in areas like homework help or medical calculations, making smart tech more accessible.
Abstract
Recent advancements in reasoning language models have demonstrated remarkable performance in complex tasks, but their extended chain-of-thought reasoning process increases inference overhead. While quantization has been widely adopted to reduce the inference cost of large language models, its impact on reasoning models remains understudied. In this study, we conduct the first systematic study on quantized reasoning models, evaluating the open-sourced DeepSeek-R1-Distilled Qwen and LLaMA families ranging from 1.5B to 70B parameters, and QwQ-32B. Our investigation covers weight, KV cache, and activation quantization using state-of-the-art algorithms at varying bit-widths, with extensive evaluation across mathematical (AIME, MATH-500), scientific (GPQA), and programming (LiveCodeBench) reasoning benchmarks. Our findings reveal that while lossless quantization can be achieved with W8A8 or W4A16 quantization, lower bit-widths introduce significant accuracy risks. We further identify model size, model origin, and task difficulty as critical determinants of performance. Contrary to expectations, quantized models do not exhibit increased output lengths. In addition, strategically scaling the model sizes or reasoning steps can effectively enhance the performance. All quantized models and codes will be open-sourced in https://github.com/ruikangliu/Quantized-Reasoning-Models.