Quantization

Quantization is the compression of a neural network by storing its weights, and sometimes its activations, at lower numerical precision. A model trained in 16-bit floating point can be converted to 8-bit or 4-bit integers, which cuts memory sharply and lets more of the model sit in fast GPU memory. Since decoding in a language model is limited mainly by memory bandwidth, smaller weights usually mean faster generation as well as a smaller footprint. The trade is accuracy. Careful methods pick per-group scaling factors and protect the weights that matter most, so quality loss at 8-bit is often negligible and at 4-bit is modest, while cruder 2-bit and 3-bit settings degrade noticeably. GPTQ and AWQ are common post-training methods, GGUF is the file format that llama.cpp and Ollama use to run models on laptops, and QLoRA applies the idea during fine-tuning so a large model can be adapted on a single GPU. The usual mistake is comparing a quantized model to its full-precision version on a benchmark alone, since degradation often shows up first in long-context reasoning and in less common languages.