RedLectern topic
Human and AI feedback · preference optimization · reward models · verifiable rewards.
For: Technical learners · researchers.
After this page you can: Distinguish preference-based methods from verifiable-reward training.
Page reviewed
Part of: Artificial Intelligence › AI training and adaptation
Preference and reward training studies how feedback changes a model after pretraining. Preference-based methods use judgments about which response is better. In reinforcement learning from human feedback, annotators compare responses, a reward model learns from those preferences, and reinforcement learning optimizes the model against that learned signal. AI feedback can supply the comparisons instead of human judgments.
Direct preference optimization trains on preferred and dispreferred responses without an explicit reinforcement-learning loop. Verifiable-reward training uses a different signal: a checker can test an answer or run unit tests, then use the result as reward. The key distinction is the source of the training signal. Preferences express a judgment about quality; verifiers check a defined criterion. This page introduces both approaches and their shared concern: the model is optimized against a signal that stands in for the intended goal.
Read the Stanford introduction to see how response comparisons lead to reward modeling, reinforcement learning, and direct preference optimization.
Start with: Foundations — Machine Learning from Human Preferences
Read a research paper describing reward signals from answer checkers, validators, and code unit tests.
Start with: Before the Model Learns the Bug: Fuzzing RLVR Verifiers
Preference-based training begins with comparisons: for a prompt, a person or another feedback source indicates which of two responses is better. In a common RLHF setup, a reward model learns to predict those preferences, then reinforcement learning updates the policy to increase the learned reward while remaining near a reference model. The reward model is a proxy for the judgments represented in its training data.
RLAIF changes who supplies the comparisons: an AI system evaluates responses and its preferences train a preference model. DPO uses preference pairs directly and removes the explicit reward-model-plus-RL pipeline described in standard RLHF. These methods share preference data, but their optimization procedures differ. A learned score or preference remains a proxy; it should not be mistaken for a direct check of correctness.
Describes training a preference model from AI-generated preferences and using it as a reward signal.
Introduces DPO and contrasts it with the explicit reward-model and reinforcement-learning steps of RLHF.
RLHF
Reinforcement learning in which rewards are derived from human feedback.
RLAIF
Reinforcement learning in which feedback supplied by AI systems provides the preference signal.
Reward model
A model that scores outputs according to a learned preference or objective.
DPO
A method that optimizes preferred responses over rejected responses without an explicit reinforcement-learning loop.
RLVR
Reinforcement learning that uses rewards checked against defined criteria, such as matching a correct answer or passing code unit tests.
A reviewer picks the clearer of two answers; a test harness checks whether a generated function passes specified tests. Which signal is preference feedback, and which is a verifiable reward?
The reviewer's choice is preference feedback because it ranks responses by judgment. The test result is a verifiable reward because it checks the function against specified tests.
Preference data records a choice among candidate outputs, often as a pairwise comparison. It can come from human annotators, or from an AI system acting as an evaluator. In the RLHF example described by Ouyang and colleagues, human labelers rank model outputs, and those rankings are used in later reinforcement learning. In Constitutional AI, a model evaluates two samples, and the resulting AI preferences train a preference model.
The comparison tells the training process which candidate was preferred under the task and evaluation criteria. It does not by itself provide an objective correctness check. Consider what the annotators or evaluator were asked to value, how consistent their choices are, and whether the preference represents the goal you intend to optimize.
Describes collecting rankings of model outputs from human labelers.
For a writing assistant, name one preference question a human reviewer could answer and one property a simple automatic checker could verify.
A reviewer could choose which of two explanations is clearer. A checker could verify whether the response contains a required phrase. The first is a judgment between responses; the second tests a defined condition.
Verifiable-reward training uses a checker to turn a defined criterion into a reward signal. Examples in recent research include answer checkers for mathematics, validators for structured tool calls, and unit-test harnesses for code. This makes the signal different from a preference judgment: the checker returns a result for a specified condition rather than choosing which response seems better.
The check only supports the criterion it actually tests. A unit-test result does not establish that code meets every possible requirement, and a correct final answer does not certify every reasoning step. This approach fits tasks with outcomes that can be checked programmatically; open-ended qualities such as clarity require a different kind of judgment or an explicitly defined scoring process.
Gives examples of verifiable reward signals for code generation and mathematical reasoning.
Investigates RLVR on mathematical and coding tasks and discusses rewards based on answer correctness.
A training task rewards a response when its final math answer matches a supplied answer key. Is that preference feedback or a verifiable reward? What does the reward establish?
It is a verifiable reward because the answer is checked against a supplied key. It establishes that the final answer matches that key; it does not establish that every reasoning step was sound.
No recent updates are available.
Johannes Fürnkranz and Eyke Hüllermeier (eds.) · 2010
A broad research treatment of preference models and ranking tasks, useful for grounding pairwise preference data. Its scope is foundational rather than a guide to modern LLM post-training.