RedLectern topic
Choose between pretraining · continued training · fine-tuning · reward-based adaptation.
For: Technical learners · builders.
After this page you can: Choose an adaptation route for a stated objective.
Page reviewed
Part of: Artificial Intelligence
AI model training covers several routes that serve different objectives. Pretraining builds broad capabilities from a training objective and a large corpus. Continued training starts from a pretrained model and trains further on domain or task data. Fine-tuning uses examples of a desired task or behavior to adapt a model. Preference and reward-based adaptation uses comparisons or other feedback to shape outputs toward a goal. These routes differ in what data they need and what behavior they directly optimize. Gururangan et al. study additional in-domain and task-adaptive pretraining, while Ouyang et al. describe supervised fine-tuning followed by learning from ranked outputs.
For the practical question of training vs fine-tuning, start with the objective: broad capability from a new model, domain exposure from more unlabeled text, task behavior from input-output examples, or preferences expressed through rankings or rewards. These are useful distinctions, not guarantees: a method only trains toward the signal and data supplied. Fine-tuning a pretrained model is a downstream adaptation route; continued pretraining keeps training with a pretraining objective, and reward-based routes optimize against feedback or an estimated reward. The sections below help you select what to investigate before choosing data, compute, or an implementation.
Start with the three-stage example of demonstrations, ranked outputs, and feedback-based optimization.
Start with: Training language models to follow instructions with human feedback
Read about continuing pretraining on domain or task text before selecting a task-specific route.
Start with: Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
Before choosing a method, write down the behavior you want to change and the evidence available to train toward it. If the goal is broad capability, investigate pretraining. If you have relevant unlabeled domain text, investigate continued pretraining. If you can provide input-output examples of the target task, investigate supervised fine-tuning. If the goal is relative quality or preference and you can gather comparisons or define a reward signal, investigate preference or reward-based adaptation. These are route-selection heuristics: the cited studies describe examples of the signals and procedures, not a universal guarantee that one route will work for every objective. Ouyang et al. describe demonstrations, rankings, and a reward model in one adaptation process.
A research example of a pretrained model adapted to downstream tasks.
You have unlabeled documents from a specialist domain and a small set of prompt-answer examples for one task. Which route would you investigate for each dataset, and what objective does each route target?
Investigate continued, domain-adaptive pretraining for the unlabeled domain documents, then supervised fine-tuning for the prompt-answer task examples. The first continues pretraining on domain text; the second trains from demonstrations of the desired behavior.
Pretraining is the broad initial training stage in which a model learns from a general corpus under a chosen training objective. Continued training begins with an existing pretrained model and performs another phase of training, often with domain or task data. In a study spanning four domains and eight classification tasks, Gururangan and colleagues report gains from domain-adaptive pretraining and further improvement from task-adaptive pretraining on unlabeled task data. That result supports considering continued training when you have relevant unlabeled text; it does not establish that it will help every model, corpus, or objective. Make the data match the target domain or task, and evaluate the resulting model on the target behavior. Read the study.
An example linking pretraining with later fine-tuning for downstream tasks.
A team has a large collection of unlabeled text from its target domain. It wants a model to better handle that domain before training a classifier. Which route is a plausible candidate, and what evidence would you use to decide whether it helped?
Domain-adaptive continued pretraining is a plausible candidate because it continues pretraining on in-domain text. Compare the resulting classifier against an appropriate baseline on the target task; the cited study reports gains across its tested domains and tasks, not a universal result.
Fine-tuning adapts an existing pretrained model to a more specific task or behavior. When you can provide examples of inputs and desired outputs, supervised fine-tuning trains the model against those demonstrations. The BERT paper describes adding a task output layer and fine-tuning the pretrained model for tasks such as question answering and language inference. In a different setting, Ouyang and colleagues describe fine-tuning on labeler-written demonstrations before using comparisons between outputs. These examples illustrate why the training signal matters: examples teach the model to imitate demonstrated outputs, so inspect whether the examples represent the behavior and cases you actually want. BERT and Ouyang et al. provide the research examples.
See how demonstrations are used for supervised fine-tuning in an instruction-following study.
You have 200 examples pairing customer messages with the replies you want the model to give. Which adaptation route directly uses these pairs, and what should a good training example contain?
Supervised fine-tuning directly uses input-output demonstrations. Each example should pair a representative customer message with a reply that illustrates the target behavior.
Use preference or reward-based adaptation when the target is expressed as comparative judgments or a reward signal rather than only a single reference answer for each input. In the InstructGPT study, researchers collect demonstrations, gather rankings of model outputs, train a reward model to predict labeler preferences, and use it to further fine-tune the model with reinforcement learning. Direct Preference Optimization proposes a different route: optimize from preference data with a classification loss, without first fitting a separate reward model. Both are research methods, and choosing between them depends on the feedback and training procedure available. Rankings express relative preferences; they do not by themselves establish an objective measure of quality. Ouyang et al. and Rafailov et al. describe these approaches.
A research alternative that trains from preferences with a classification loss.
For the same prompt, annotators can rank two candidate replies but cannot agree on one canonical reply. Which family of routes fits that feedback, and what does the ranking provide?
Preference-based adaptation fits comparative rankings. The ranking indicates which of the two replies annotators prefer; in the InstructGPT procedure, such rankings train a reward model that is then used as a reward signal.
Task and domain adaptation; supervised tuning; efficient adapters; distillation.
0 subtopicsPreference and reward trainingHuman and AI feedback; preference optimization; reward models; verifiable rewards.
0 subtopicsPretrainingBroad initial training; objectives; data and compute allocation; scaling evidence.
0 subtopicsNo recent updates are available.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville · 2016
The authors provide the complete online text free. Its chapters on training deep models and representation learning give builders foundations for understanding how models are trained and adapted.
Richard S. Sutton and Andrew G. Barto · 2018
A free online copy is linked from a University of Wisconsin course page. The book develops the foundations of learning from interaction and reward, useful context for reward-based adaptation.
Christopher M. Bishop · 2006
This hosted reference develops machine learning from a Bayesian perspective. Use it to examine the assumptions behind candidate learning methods when choosing a training signal.
Tom M. Mitchell · 1997
The hosted edition introduces machine learning across neural networks, Bayesian learning, and reinforcement learning. Use it to frame the learning problem before choosing an adaptation route.