Quick answer: Quantization is the process of shrinking the numbers inside a trained model, its weights and often its activations, from a high precision format such as 32 bit floating point down to a lower precision format such as 8 bit or 4 bit integers. It cuts a model’s memory footprint and speeds up inference, frequently by four times or more, at some cost in accuracy. Quantization is a major reason a language model that once needed a data center GPU can now run on a phone or a laptop.
So what does quantizing a model actually change? It rounds each number onto a much coarser scale, the way a photo saved with fewer colors still looks recognizable even though it loses fine gradients. This entry covers how quantization works mechanically, where you meet it in tools you already use, how it compares to model pruning, and where it runs into trouble.
What Is Quantization?
A model fresh out of training stores its weights as 32 bit or 16 bit floating point numbers, a format that gives fine grained precision but takes up a lot of memory and needs floating point hardware to compute quickly. Quantization maps those numbers onto a smaller, fixed set of values, commonly signed 8 bit integers or 4 bit integers, using a scale factor that translates between the original range and the new, coarser one.
The compression is roughly proportional to the bit width. Dropping from 32 bit to 8 bit numbers cuts memory by about four times, and dropping to 4 bit cuts it further still, though the loss of precision grows as the numbers get coarser. Some quantization schemes only touch a model’s weights, leaving activations in higher precision; others quantize both weights and activations for extra speed on hardware built for integer math.
Quantization schemes also differ in how finely they adapt to a model’s actual value range. A single scale factor for an entire layer is simple but wastes precision if some parts of that layer use a much narrower range than others, which is why many quantization tools calculate a separate scale for each channel or block instead, trading a little extra bookkeeping for a tighter fit and less accuracy loss.
How Does Quantization Work?
There are two broad ways to get there. Post training quantization converts an already trained model using a small calibration dataset to measure the typical range of each layer’s values, then rounds accordingly. It is fast, often taking minutes, and needs no retraining.
Quantization aware training instead simulates the rounding during training itself, so the model’s weights adjust around the precision loss before it ever happens. It costs a full training run, but it holds up better at very low bit widths, where post training quantization alone tends to lose more accuracy.
In practice, most people meet quantization through a file format rather than the underlying math. Tools built on llama.cpp save models in GGUF files at specific quantization levels, such as Q4_K_M, which blends 4 bit and higher precision blocks to balance size against quality. Other toolchains, including GPTQ and AWQ, apply their own calibration strategies to reach similarly small footprints.
Most of these local model formats quantize only the weights, since a language model’s activations tend to have a wider and less predictable range than its weights, and forcing both onto the same coarse scale causes more damage than it saves in speed.
Where Is Quantization Used?
The clearest example is running an open language model locally. A Llama family model quantized to Q4_K_M shrinks to roughly a quarter of its original size with only a modest quality loss, which is what lets a 7 or 8 billion parameter model run comfortably on a laptop instead of a server rack.
Quantization is not only a mobile story either. Serving a model at INT8 instead of full precision on standard data center GPUs lowers the memory and cost of running it at scale, which is part of why a quantized version of a major open model is usually available alongside the original full precision release.
Phone and laptop chipmakers build quantization support directly into their hardware. Apple’s on-device stack quantizes, and in some cases palettizes, models so they run efficiently on the NPU inside recent iPhone and Mac chips. Apple has described its own on-device foundation model as using low-bit palettization in a mixed 2 bit and 4 bit configuration, averaging 3.7 bits per weight, with small adapter layers trained on top for individual tasks.
The same pattern shows up across edge AI generally, where a quantized model is often the only version small enough to fit a device’s memory at all.
That is also why quantization sits behind so much of the recent interest in small local models. A small language model that would already be modest in size becomes genuinely lightweight once quantized, which is a large part of why compact, offline capable assistants have become practical on ordinary consumer hardware rather than staying a server side product.
Quantization vs Model Pruning: What’s the Difference?
Both are compression techniques aimed at the same goal, a smaller and faster model, but they act on different things. Quantization changes how each number is represented, packing the same count of weights into fewer bits apiece. Model pruning instead removes numbers outright, deleting weights, neurons, or entire layers that contribute little to the model’s output.
The two are not competitors. A model is commonly pruned first to cut its parameter count, then quantized on top of that to shrink each remaining parameter further, since the two techniques target different sources of size.
What Are the Limits of Quantization?
Accuracy loss is not linear with bit width. Going from 16 bit to 8 bit is usually close to free, but pushing to 4 bit or lower, especially without quantization aware training, can produce a noticeable drop on tasks that need precise reasoning or rare, specific outputs.
Large language models make this harder in one specific way: a small number of activation values tend to run far larger than the rest of a layer, and naive quantization that ignores those outliers introduces more error than the typical case would suggest. Several quantization methods handle this by treating outlier values separately from the bulk of a layer’s numbers instead of forcing everything onto one shared scale.
Hardware support is uneven too. A quantization format tuned for one chip’s integer instructions does not necessarily run efficiently on another, so the theoretical size and speed benefits depend on running the model on hardware built to exploit that particular format.
None of that has slowed adoption. Quantization is now close to a default step in shipping a model outside a data center, and it is the main reason the gap between a cloud model and a model that fits on your device keeps narrowing.