Preference and reward training concept
What is RLHF?
ReferenceWhat is RLHF? Reinforcement learning from human feedback is a way to fine-tune a model using people’s preferences as a learning signal. In a common language-model workflow, people compare responses, a reward model learns to predict those preferences, and reinforcement learning adjusts the language model using that learned score. The method can steer behavior toward the preferences represented in its feedback data, but those preferences are not a universal measure of what people want.
Reviewed 2026-10-09
A training method built around preferences
RLHF means reinforcement learning from human feedback. It is a way to train a model using judgments from people, especially when the desired behavior is hard to describe with a simple automatic score. The feedback can express which of two outputs a person prefers. The model is then trained to produce outputs that receive higher scores based on those preferences.
For language models, RLHF is usually part of fine-tuning: it changes a model’s behavior after an earlier training stage. The feedback teaches the system what its annotators preferred in the examples they saw. It does not define a single, universal human preference.
From comparisons to a learned reward
One commonly described language-model workflow begins with supervised fine-tuning, where a model learns from examples of responses people want. Next, people compare several model responses to the same prompt. A reward model is trained on those comparisons to estimate which responses the labelers would prefer.
The language model is then optimized using the reward model’s scores. In the InstructGPT paper’s version of this workflow, the team used proximal policy optimization, a reinforcement-learning algorithm, for that step. This means the human judgments usually guide training through a learned scoring model; people do not need to rate every response generated during the optimization step.
One route through preference training
RLHF sits within preference and reward training: comparisons provide preference data, a reward model can turn those comparisons into scores, and reinforcement learning can optimize a model against those scores. The broader idea of reinforcement learning is learning by using rewards to guide an agent’s actions.
The term is often used for a pipeline that explicitly trains a reward model and then uses reinforcement learning. Direct Preference Optimization (DPO) is a related method: its authors describe it as solving the standard RLHF problem with a classification loss. So, when someone says “RLHF,” it helps to check whether they mean the broad use of human preferences or that particular reward-model-and-RL pipeline.
What the feedback does and does not mean
“Human feedback” does not necessarily mean every end user rates every answer. In the InstructGPT workflow, labelers provided demonstrations and compared model outputs; the resulting preference data trained a reward model. The people and instructions behind the feedback shape what that model learns to reward.
A high reward-model score means the response fits the learned preference signal, not that it is objectively correct or represents everyone’s values. RLHF is a training approach for steering behavior, not a guarantee of truth or safety. The InstructGPT paper itself notes that the resulting models still made simple mistakes.
Questions people ask
Does a person have to rate every answer during RLHF?
No. In the InstructGPT workflow, people supplied demonstrations and ranked selected model outputs; a learned reward model then scored responses during reinforcement-learning fine-tuning.
Does RLHF make an AI model truthful?
No. It can train a model toward preferences that include truthfulness, but it does not guarantee correctness. The InstructGPT paper reports that its models still made simple mistakes.
Does RLHF only apply to language models?
No. Earlier work studied human preference feedback for reinforcement-learning tasks such as Atari games and simulated robot locomotion. The language-model workflow is one application.
Is DPO the same as RLHF?
They are related, but the terms are not interchangeable in every use. The DPO paper presents its method as solving the standard RLHF problem with a classification loss, instead of the usual separate reward-model and reinforcement-learning steps.
Learn more
- Training language models to follow instructions with human feedback
Read the paper that lays out the demonstrations, preference comparisons, reward model, and reinforcement-learning stages for InstructGPT.
- Deep reinforcement learning from human preferences
See an earlier example of using human comparisons to define goals for reinforcement-learning tasks beyond language models.
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Compare a preference-optimization method that the authors frame as solving the standard RLHF problem using a classification loss.
What changed
Sources
Part of Preference and reward training: Feedback, reward models, and optimizationWhat preference data represents