Fine-tuning concept
What is LoRA?
ReferenceWhat is LoRA? LoRA, or Low-Rank Adaptation, is a way to fine-tune a pretrained neural network by training a small set of added parameters while keeping its original weights fixed. It represents a weight change with two smaller matrices, which can take fewer trainable parameters than updating the full weight matrix. LoRA is one form of parameter-efficient fine-tuning; QLoRA combines it with a quantized base model.
Reviewed 2026-10-09
A small learned change to a pretrained model
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: it adapts a pretrained neural network by adding small trainable matrices while leaving the original weights fixed. Fine-tuning means further training a pretrained model toward a task or domain. For a layer with a weight matrix, LoRA represents the learned change as the product of two smaller matrices instead of training a full-sized change to that matrix. The size of this factorization is called its rank. LoRA is often described as an adapter, meaning a set of learned additions used with a base model.
How the update is learned
During LoRA fine-tuning, the base model’s weights stay frozen, while training updates the added low-rank matrices. Their product represents the change to a selected weight matrix. The rank controls the dimensions of these matrices: a higher rank adds more trainable parameters and gives the update more learning capacity. Which weight matrices receive LoRA additions is a design choice; common implementations target attention blocks, but LoRA can be applied to other selected weights too. After training, the additions can be used with the base model or merged into it.
One option within fine-tuning
Full fine-tuning updates all of a model’s parameters. LoRA is a form of parameter-efficient fine-tuning, which trains a smaller set of added parameters while the base weights remain fixed. It can therefore reduce the number of parameters that need training, but the result still depends on the task, data, and training choices. LoRA is commonly applied to attention blocks in Transformer models, though it is not limited to them. In the Fine-tuning topic, it belongs under Choose an adaptation method.
LoRA, adapters, and QLoRA
LoRA is an adapter-style method, but “adapter” can also mean other added components used to adapt a model. The useful distinction is how the update is represented: LoRA uses a low-rank factorization. A LoRA file is an addition to a base model; it can be merged into that model to make a standalone merged model. QLoRA is related but distinct: it trains LoRA additions while gradients pass through a frozen, quantized base model. Quantization represents model weights with fewer bits; in the QLoRA paper’s approach, the base model is quantized to 4 bits. That quantized base is what distinguishes QLoRA from ordinary LoRA.
Questions people ask
Why does LoRA use two smaller matrices?
Their product represents a weight update in a low-rank form. This lets training learn a smaller set of parameters than a full-sized update to that weight matrix.
Does LoRA only work for language models?
No. LoRA was first developed for large language models, and it is also used to train diffusion models.
What changes in QLoRA?
QLoRA keeps the LoRA-style trainable additions but sends gradients through a frozen, quantized base model. The QLoRA paper describes a 4-bit quantized base.
Learn more
- LoRA: Low-Rank Adaptation of Large Language Models
The original paper explains the frozen base weights and trainable rank-decomposition matrices.
- LoRA
Explains the matrix factorization, rank, common target layers, and merging LoRA weights into a base model.
- QLoRA: Efficient Finetuning of Quantized LLMs
Shows how QLoRA combines LoRA adapters with a frozen, 4-bit quantized pretrained model.
What changed
Sources
Part of Fine-tuning: Choose an adaptation method