Training Dynamics Impact Post-Training Quantization Robustness
Albert Catalan-Tatjer, Niccolò Ajroldi, Jonas Geiping
2025-10-08
Summary
This research investigates why some large language models handle a process called 'quantization' better than others, and what can be done to improve how well they perform after quantization.
What's the problem?
Large language models are huge and require a lot of computing power. 'Quantization' is a technique to make them smaller and faster, but it often reduces their accuracy. The researchers wanted to understand *why* quantization causes accuracy loss, and if it's simply unavoidable as models get larger and are trained on more data. They suspected the way the models are trained plays a big role, but this hadn't been thoroughly investigated.
What's the solution?
The team analyzed how accuracy changed during the training of several large language models, looking specifically at the relationship between training settings like the 'learning rate' (which controls how quickly the model learns) and the amount of error introduced by quantization. They found that once the learning rate starts to decrease during training, the model's performance on unseen data and the error from quantization start to diverge. To test this further, they trained their own models with different settings, carefully controlling the training process up to a massive 100 billion tokens of text.
Why it matters?
This work shows that simply using more training data doesn't automatically mean a model will be less effective after quantization. Instead, carefully choosing training settings, particularly how the learning rate changes over time, can actually *improve* how well a large language model performs after being made smaller and faster with quantization. This is important because it means we can potentially deploy powerful language models on less expensive hardware without sacrificing too much accuracy.
Abstract
While post-training quantization is widely adopted for efficient deployment of large language models, the mechanisms underlying quantization robustness remain unclear. We conduct a comprehensive analysis of quantization degradation across open-source language model training trajectories up to 32B parameters and 15T training tokens to accurately assess the relationship between training dynamics and quantization performance. Our key finding is that quantization errors in large-scale training runs are driven by a complex interplay between learning rate and other training hyperparameters. Specifically, once learning rates decay, validation loss and quantization error diverge, largely independent of training data scale. To investigate interventions on the training dynamics and identify specific configurations that can modulate quantization robustness favorably, we train our own models in controlled experiments up to 100B tokens. Our results challenge the assumption that increasing dataset scale inherently compromises quantization effectiveness, demonstrating instead that strategic training hyperparameter interventions can improve quantization quality at scale.