Fine-tuning concept
LoRA versus QLoRA
ReferenceLoRA vs QLoRA: both adapt a pretrained model by training small low-rank adapters while keeping its main weights frozen. QLoRA adds quantization: it stores the frozen base weights using fewer bits, commonly four, then trains the adapters through that quantized base. This can reduce memory use, but the quantized weights may affect results. The practical choice depends on available memory and how each method performs on your task.
Reviewed 2026-10-09
Two ways to adapt a pretrained model
LoRA means Low-Rank Adaptation. It freezes the pretrained model’s weights and learns small, trainable matrices that represent changes to selected layers. Those matrices are called adapters; they let training adjust the model without updating all its original weights.
QLoRA means quantized LoRA. Quantization represents weights with fewer bits, reducing the memory they take up. In QLoRA, the base model’s weights are frozen in a quantized form, while LoRA adapters are trained on top. The name can sound like a separate kind of adapter, but its central distinction is how the frozen base weights are stored during training.
The main difference is the base weights
Both methods train adapters rather than changing all the base model’s weights. In the original LoRA approach, the pretrained weights stay frozen and the low-rank matrices are trainable. QLoRA keeps that adapter approach but quantizes the frozen base weights; the paper describes gradients passing through a frozen, four-bit quantized model into LoRA adapters.
That lower-bit representation is why QLoRA can use less memory for the base weights. It also means the two approaches do not use identical weight representations during training. Quantization can affect results, so a smaller memory footprint should not be taken to mean that quality is unchanged. The paper’s memory and performance findings describe its experiments; they do not guarantee the same outcome for every model, dataset, or task.
Choose according to memory and results
Consider LoRA when the base model fits within your available training memory and you want to avoid quantizing its weights. Consider QLoRA when memory is the constraint and training from a quantized base could make the experiment practical. This is a decision rule drawn from the methods’ difference and QLoRA’s memory-reduction goal, not a claim that one method always performs better.
For either method, make the comparison on your own task: use the same held-out examples and scoring procedure, and check the adapted model against the untouched base. If QLoRA saves enough memory to make the run possible, compare its results with LoRA when you can afford both runs. A memory saving alone does not tell you which adapted model better meets your task’s needs.
What the names do and do not imply
QLoRA is not full fine-tuning of a four-bit model: in the method described by its authors, the base weights are frozen and the LoRA adapters are trained. LoRA is not synonymous with keeping every weight at one particular precision; the central idea is freezing pretrained weights and training low-rank changes. The comparison is about the adapter method and the base-weight representation used during training.
Four-bit quantization does not mean that every part of training uses four-bit arithmetic. The QLoRA paper describes a four-bit quantized base and separately reports its training approach; implementation choices can specify a compute data type. Nor does quantization guarantee unchanged quality. Treat the memory reduction as a practical benefit to test alongside task results.
Questions people ask
What does the Q in QLoRA mean?
It refers to quantization: QLoRA stores the frozen base weights in a lower-bit representation and trains LoRA adapters on top.
Does QLoRA train the base model’s weights?
In the QLoRA method described by its authors, the base weights are frozen. Gradients pass through the quantized base to train the LoRA adapters.
Does QLoRA always produce the same quality as LoRA?
No such guarantee follows from the method. Quantization changes how base weights are represented, and results can depend on the model and task. Compare them on held-out examples relevant to your use.
Can LoRA be used with a quantized model?
Yes. LoRA is the adapter method, and official PEFT documentation describes adding LoRA adapters to quantized models. QLoRA is a particular approach that combines a quantized base with LoRA training.
Learn more
- LoRA: Low-Rank Adaptation of Large Language Models
Read the original description of freezing pretrained weights and training low-rank matrices.
- QLoRA: Efficient Finetuning of Quantized LLMs
See how the authors describe a quantized frozen base, trainable LoRA adapters, and memory-saving techniques.
- Quantization
See a practical guide to quantizing a base model and training LoRA adapters on it.
What changed
Sources
Part of Fine-tuning: Choose an adaptation methodPrepare a small experimentEvaluate the result