Model quantization concept

What is model quantization?

Reference

What is model quantization? It is a way to represent a model’s numbers with fewer bits, often to reduce the storage needed for its weights and the memory needed to use it. The conversion approximates the original values, so some quality loss is possible, but its size depends on the method and the model’s task. Quantization describes a numerical representation; it does not by itself specify a file format or guarantee faster inference.

Reviewed 2026-10-09

What quantization means

Model quantization changes how a model’s numerical values are represented. In a language model, weights are learned numbers used to transform inputs, while activations are intermediate values produced as the model processes an input. Quantization stores one or both using a representation with fewer possible values, such as a lower-precision number type. The number of bits used to represent a value is its bit width. Fewer bits can reduce the storage required for those values. Because the new representation approximates the old one, quantization can introduce small numerical errors; the aim is to keep the model’s performance on its task from changing too much.

How the conversion works

A quantization method maps a range of original values to a smaller set of representable values. In a common approach, it uses a scale to relate the two ranges and may use a zero-point to align them; values are rounded to the nearest available representation. During inference, calculations can use the quantized values, sometimes converting results back to a higher-precision representation between operations. Methods differ in what they quantize and how they choose the mapping. Some use calibration, which means examining sample inputs to help set the mapping. The choice of method and granularity affects the errors introduced, so “4-bit” or “8-bit” alone does not tell you the expected quality.

Where it fits among model choices

Quantization is one way to reduce the resources used to store or run a model. Post-training quantization is applied after training is complete. Quantization-aware training simulates quantization effects during training so the model can adapt to them. These are different points in the training process, not names for file formats. A quantization label describes a numerical encoding choice; the file format describes how model data is stored, and software support determines whether a particular combination can be loaded and run. Check the method, format, and software compatibility together when choosing a quantized model.

What quantization does not tell you

Quantization can reduce accuracy, but the change is not fixed by bit width alone. Results depend on the method, what values are quantized, and the task; some methods are designed to retain performance while lowering precision. A smaller weight representation also does not mean every part of inference uses less memory: activations and, for language models, the KV cache are separate working values. Nor does a smaller representation guarantee faster inference. Speed depends on whether the software and hardware support efficient operations for that representation. Test the quantized model on the tasks and setup that matter to you.

Questions people ask

Does quantization reduce accuracy?

It can, because representing values with fewer possible numbers introduces approximation error. The amount of change depends on the quantization method, the values it affects, and the task; it is not determined by bit width alone.

Does quantization make a model faster?

It can, but a smaller representation does not guarantee faster inference. Speed depends on whether the software and hardware can perform efficient operations on that representation.

What is the difference between post-training quantization and quantization-aware training?

Post-training quantization is applied after training is complete. Quantization-aware training simulates quantization effects during training, so the model can adapt to them.

Does a quantization label tell me the model’s file format?

No. The label describes a numerical encoding choice. The file format describes how model data is stored, and software compatibility determines whether it can be loaded and run.

Learn more

Quantization concepts
Explains value mapping, quantization methods, and the accuracy tradeoff.
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Shows how quantization can be paired with training to preserve model accuracy after conversion.

What changed

Sources

Part of Model quantization: What fewer bits change Test the quality tradeoff Check the file format and runtime