Quantization overview
Start with the precision tradeoff
Read the overview, then compare the lower-precision representation and its accuracy tradeoff.
Start with: Quantization overview
RedLectern topic
Model quantization
Reduced numerical precision · memory savings · quality tradeoffs · compatible formats.
For: Local users · technical builders.
After this page you can: Choose a quantized model and test its quality tradeoff.
Page reviewed
Part of: Artificial Intelligence › Local and on-device AI
About Model quantization
Quantization represents model values with fewer numerical bits. For model weights, fewer bits can reduce the storage needed for those weights, which can make a model easier to fit into a constrained setup. The tradeoff is that approximating the original values can affect model quality; the effect depends on the model, quantization method, and task.
Choosing a quantized model is therefore a practical comparison. Start with a format your intended runtime supports, then compare its outputs with a reference model on representative prompts and tasks. Keep the prompts, settings, and scoring criteria consistent, and check both the answers and any errors that matter for your use. Quantization describes a model representation, not where inference runs; a local AI app can still use cloud models, so inspect its data path.
Start here
Scaling Laws for Post Training Quantized Large Language Models
Plan a quality check
Use the same representative tasks to compare a quantized model against a reference.
Start with: Scaling Laws for Post Training Quantized Large Language Models
Learn the foundations
1.What fewer bits changeQuantization maps values to a representation with fewer bits.
Quantization maps values to a representation with fewer bits. For model weights, reducing precision can reduce the storage needed for those weights. Bit width is one part of the decision: FP16 and BF16 are 16-bit floating-point formats, while INT4 is a four-bit integer representation. BF16 uses more exponent bits than FP16, giving it a wider exponent range.
Post-training quantization applies this conversion after the original training. A named preset such as Q4_K_M refers to a particular quantization choice in llama.cpp; the label alone does not tell you how your tasks will fare. Compare the actual model outputs before choosing.
- Tensor Encoding Schemes
A reference for how selected tensor encodings use bits and blocks.
Quantization
Representing model values with fewer numerical bits.
Bit width
The number of bits used to represent a value.
FP16
A 16-bit floating-point numerical format.
BF16
A 16-bit floating-point format with a wider exponent range than FP16.
INT4
A four-bit integer representation used in some quantized models.
Post-training quantization
Quantizing a model after its original training.
Q4_K_M
A mixed-precision preset in llama.cpp's four-bit K-quant family.
Try it
A model has a choice of 16-bit floating-point weights or 4-bit integer weights. Which uses fewer bits per weight, and what should you compare before choosing the 4-bit version?
The 4-bit representation uses fewer bits per weight. Compare both versions on representative tasks because lower precision can affect accuracy; also confirm that the intended runtime supports the chosen model format.
2.Test the quality tradeoffLower-precision weights can save storage, but the quality change is not a fixed property of a bit width.
Lower-precision weights can save storage, but the quality change is not a fixed property of a bit width. Research on post-training quantization reports that quality can be difficult to predict and may require case-by-case validation. A useful test starts with tasks you actually care about, then compares a quantized model with a reference using the same prompts and evaluation conditions.
Record failures as well as successful answers. For example, check whether summaries omit key points, extraction misses fields, or answers become less reliable on your own task set. The comparison helps you decide whether the storage benefit is worth the observed quality change for your use.
- ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
Studies memory reduction and the accuracy tradeoff in post-training quantization.
Try it
Choose one task you use a model for. Write two prompts that represent it, then name one answer feature you would compare between a reference and a quantized model.
For a summarization task, use two representative passages and compare whether each version retains the same key points without adding unsupported details.
3.Check the file format and runtimeA quantization label and a model file format answer different questions.
A quantization label and a model file format answer different questions. The label describes a numerical encoding choice; the file format determines how model data and metadata are stored. GGUF is a file format used for models intended for inference with GGML and executors based on GGML.
Compatibility depends on the model file and the software that loads it. Before choosing a download, check that your intended runtime supports the file format, model architecture, and quantization type together. Treat this as a compatibility check for the specific files and software versions you plan to use; a familiar quantization label by itself is not enough.
- Quantization overview
Points readers to method support and hardware considerations for a chosen quantization approach.
Try it
A model page lists a quantization label but does not say which file format it uses. Name two compatibility details to verify before downloading it.
Verify the file format and whether the intended runtime supports the model architecture and quantization type in that format.
Latest
No recent updates are available.
Books
TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers
Pete Warden and Daniel Situnayake · 2019
A practical companion for getting TensorFlow Lite models onto constrained devices. It also discusses reducing model size and debugging models.