Accurate Low Bit Quantization Through Function Preserving Transformations
Quantization is a key enabler for deploying large language and multimodal models efficiently, yet reducing precision below 8 bits often leads to significant accuracy degradation. This talk presents a unified perspective on recent advances in low-bit quantization through the use of representation transformations. We will first introduce FPTQuant, which leverages function-preserving transformations to reshape intermediate representations for robust INT4 inference with negligible runtime overhead. We will then discuss a theoretical framework that explains quantization error through the interplay of representation concentration and alignment, motivating Concentration-Alignment Transforms (CAT) for improved post-training quantization. Finally, we will present STaMP, which extends the idea of transformations to the sequence dimension, exploiting token correlations together with selective mixed-precision computation to enable more accurate low-bit activation quantization. Together, these works demonstrate how lightweight transformations can substantially improve quantization quality without modifying model architectures or requiring specialized hardware, offering a principled path toward efficient deployment of foundation models across cloud, edge, and on-device platforms.