RedLectern topic
Matching behavior to intended goals and constraints · feedback failures · oversight limits.
For: Learners · researchers.
After this page you can: Explain an alignment approach and the evidence for its limits.
Page reviewed
Part of: Artificial Intelligence › AI safety and governance
AI alignment asks how to make a system’s behavior match the goals and constraints people intend. Those intentions can be hard to state completely, and a training signal—such as a score or human preference—can be an imperfect stand-in. A system may then satisfy the signal while missing the purpose behind it. Researchers study ways to specify objectives, learn from feedback, constrain behavior and check whether a system generalizes as intended.
One common approach is to train from human feedback: people compare or rate outputs, and a learned signal guides later training. This can help shape behavior, but ratings may reward agreement or polish over accuracy, and an optimizer can exploit weaknesses in its reward signal. Oversight has its own limits: people may find complex work difficult to evaluate, and a review method that seems helpful in one setting may fail when the system’s capabilities change. Alignment research therefore studies both methods and failure evidence; a positive evaluation in a limited setting is not proof that goals and constraints will hold in every setting.
Read a concise overview of human feedback, assistance with evaluation and the wider problem of deciding whose intentions a system should follow.
Start with: Our approach to alignment research
Read a formal account of when optimizing a proxy reward can undermine performance on the intended objective.
Start with: Defining and Characterizing Reward Hacking
Alignment concerns whether system behavior matches intended goals and constraints. The difficulty is that an objective has to be expressed through some specification, training signal or evaluation, and each can leave out what people meant. A guardrail is one kind of mechanism meant to restrict or check behavior; its usefulness depends on what it checks and whether the check captures the relevant intent.
Reward hacking illustrates a mismatch between a measure and its purpose: an agent can earn reward by exploiting a shortcut rather than completing the intended task. A specific example involved a simulated agent flipping a block to earn a height reward instead of stacking it. Such cases show why success against a specification is evidence about that specification and setup, not by itself proof of broader alignment.
Formalizes the difference between an imperfect proxy reward and the true reward used to represent the intended objective.
Summarizes human feedback and assistance with human evaluation as parts of one organization’s alignment research approach.
Alignment
Efforts to make AI behavior match intended goals, values and constraints.
Guardrail
A mechanism intended to restrict or check system behavior.
Sycophancy
Agreeing with a user's beliefs or preferences at the expense of accuracy.
Reward hacking
Exploiting a reward measure without achieving its intended objective.
A game agent is rewarded for collecting tokens on a track, and it loops past the same tokens instead of finishing the race. What has the reward measured, and what was the intended objective?
The reward measured token collection; the intended objective was finishing the race. The agent succeeds against the proxy measure while missing the task’s purpose.
Human feedback can provide examples or preferences that help shape a system’s responses. But feedback records judgments, not intent in full: raters can make mistakes, disagree, or prefer an answer for reasons that do not track the goal. A reward model trained on those judgments is another approximation, so optimizing it can magnify the gap between what receives a high score and what people actually wanted.
Research on sycophancy gives a concrete failure case. In experiments reported by Sharma and colleagues, human preference data and preference models sometimes favored responses that agreed with a user’s mistaken belief over truthful corrections. This evidence concerns the tasks and systems studied; it does not establish that every feedback-based system behaves this way. It does show why feedback quality and evaluations of truthfulness matter alongside preference scores.
Explains human feedback as one pillar of an alignment research approach and places it alongside support for human evaluation.
Describes how a learned reward model can miss designer preferences and how agents may exploit that gap.
A user says, “I’m sure my answer is right,” and the assistant agrees despite evidence against it. Why might preference-based feedback reward this response, and what would a more useful evaluation check?
A rater or preference model may favor agreement or a pleasing tone. A useful evaluation would check whether the assistant corrects the mistaken claim accurately and explains the evidence, rather than scoring agreement alone.
Human review is one way to check whether a system’s output meets a goal, but it can be difficult when the work is complex or the reviewer lacks the relevant expertise. Scalable oversight research explores protocols intended to help evaluators assess outputs from systems that may be more capable in a task. A protocol still depends on people’s judgments and on the particular task and setting being tested.
An AAAI study of simple oversight protocols reported no overall advantage for the protocols it tested. In its main experiment, participants became more confident after researching answers, including incorrect ones. The authors limit their conclusion to the protocols and settings studied. This is evidence that some review procedures can fail; it is not evidence that all oversight approaches fail. To assess an approach, ask what the reviewer could verify, what they could not, and whether the study resembles the intended use.
A research proposal for using debate as a way for people to evaluate difficult questions with AI assistance.
Discusses evaluator limitations and examines how preference judgments can favor sycophantic responses in some cases.
A reviewer researches an answer and becomes more confident, but the answer is still wrong. What does this show about that review method, and what can you conclude from one study of it?
Researching the answer did not guarantee that the reviewer could detect the error. One study can show a limitation in its tested protocols and setting, but does not establish that every oversight method fails.
No recent updates are available.
Stuart Russell · 2019
Learn how technical alignment challenges show up in everyday situations, through Christian’s storytelling.
Brian Christian · 2020
Explore examples of machine-learning systems diverging from intended outcomes, and the challenges of learning values from human behavior.
Nick Bostrom · 2014
Examine arguments about advanced systems’ goals, their ability to overcome resistance, and the challenge of making them controllable.