5.1 Reinforcement Learning
Start with the interaction loop
Read the introductory section on reinforcement learning, then identify the agent, actions, feedback and policy in its example.
Start with: 5.1 Reinforcement Learning
RedLectern topic
Reinforcement learning
Learning through rewards · policies · exploration · sequential decisions.
For: Technical learners
After this page you can: Describe and evaluate a simple reward-based learning problem.
Page reviewed
Part of: Artificial Intelligence › Machine Learning
About Reinforcement learning
Reinforcement learning studies how an agent can learn what to do by interacting with an environment and receiving feedback. At each step, the agent observes a situation, chooses an action, and receives a reward as the environment changes. A policy is the rule the agent uses to choose actions. The goal is usually to choose a policy that earns high cumulative reward across a sequence of decisions, rather than to maximize only the next reward.
To describe a simple problem, specify what the agent can observe, which actions it can take, how the environment responds, and what earns reward. Then ask whether the reward captures the outcome you actually want. The agent may need to explore unfamiliar actions to learn their consequences, while also using actions that currently seem promising. This creates an exploration-exploitation tradeoff. A useful first exercise is to sketch a small environment, propose a reward, and consider how a policy could behave well in the short term while doing poorly over a longer sequence.
Start here
Part 1: Key Concepts in RL
Build the formal vocabulary
Use this introduction to connect states, actions, policies, rewards and cumulative return.
Start with: Part 1: Key Concepts in RL
Learn the foundations
1.The parts of a reward-based problemA reinforcement learning problem is framed as interaction: an agent receives an observation, chooses an action, and the environment responds with a new observa…
A reinforcement learning problem is framed as interaction: an agent receives an observation, chooses an action, and the environment responds with a new observation and reward. Learning from this feedback can help the agent estimate a policy, the rule it uses to choose actions. The reward signal represents what the problem designer wants to encourage, while the environment defines the situations and responses the agent encounters.
The reward at one step is not necessarily the whole objective. Many problems evaluate the cumulative reward over a trajectory, so actions can matter because of what they lead to later. When describing a problem, state the agent’s observations and possible actions, describe how the environment responds, and say how rewards accumulate. Then check whether the chosen reward actually reflects the desired outcome.
- 5.1 Reinforcement Learning 10 min
Explains how an agent can use exploration feedback to estimate a policy.
Reinforcement learning
Learning action choices through feedback expressed as rewards.
Policy
A rule or model that specifies actions for an observed situation.
Reward
A signal expressing how desirable an action or outcome is.
Environment
The setting that responds to actions with observations and feedback.
Exploration-exploitation
The tradeoff between trying uncertain actions and using currently promising ones.
Try it
A delivery robot can take a short route with a small chance of a large delay, or a longer route with a predictable travel time. Name the agent, one possible observation, two actions, an environment response and a reward that would reflect reliable delivery.
The robot is the agent; traffic or current location could be an observation. The actions are the short and long routes. The environment returns arrival time or delay. A reward could favor on-time delivery and penalize late arrival, so reliability matters rather than distance alone.
2.Actions have consequences over timeIn a sequential problem, an action can change the environment and affect what situations the agent encounters next.
In a sequential problem, an action can change the environment and affect what situations the agent encounters next. That means a choice with a modest immediate reward may lead to better later outcomes, while a tempting immediate reward may create a worse sequence. Cumulative reward, often called return, gives a way to express the overall objective across a trajectory.
To evaluate a simple problem, trace a few possible action sequences and compare their total rewards. Be explicit about the time horizon and, if future rewards count less, how they are discounted. Also consider whether the agent sees enough of the situation to choose well: an observation may omit information about the full state. These choices affect what a policy can learn and what “good performance” means.
- Reinforcement Learning Basics 25 min
Lecture slides connect interaction and policy improvement to long-term reward.
Try it
A two-step task offers either +3 now and then 0, or 0 now and then +5. With no discounting, which sequence has higher total reward? What changes if the second reward is discounted by 0.5?
Without discounting, the first sequence totals 3 and the second totals 5, so the second is higher. With a 0.5 discount on the second reward, it is worth 2.5; the first sequence’s 3 is then higher.
3.Learn what works, then evaluate the tradeoffsAn agent may not know at first which actions lead to high reward.
An agent may not know at first which actions lead to high reward. Exploration gathers feedback about uncertain choices; exploitation uses what the agent currently believes is promising. A simple approach can deliberately choose a random action some of the time and otherwise follow its current policy. More exploration can reveal useful alternatives, but too much can keep behavior random; too little can leave better actions undiscovered.
Evaluate a reward-based problem by asking what the agent had a chance to try, how its feedback supports the learned policy, and whether success is measured across enough runs and decision sequences. A policy that performs well on one short trajectory may not be reliable. Also inspect the reward design: if it rewards an easy-to-measure proxy, the agent may optimize that proxy without achieving the broader goal.
- 5.1 Reinforcement Learning 10 min
Describes learning from successor-state and reward feedback during exploration.
- Part 1: Key Concepts in RL 30 min
Defines policy and return and explains choosing a policy by expected return.
Try it
A learner has tried two actions. Action A has produced rewards of 2, 2 and 2; action B has produced 0 once. Give one reason to choose A now and one reason to try B again.
A currently looks promising because its observed rewards were consistently higher. B may still be worth trying because one result gives little evidence about its typical reward; another trial could reveal that it is better or more variable.
Latest
No recent updates are available.
Books
Reinforcement Learning: An Introduction
Richard S. Sutton and Andrew G. Barto · 2018
A course syllabus uses this book for reinforcement learning modules covering the topic’s core material and links a free PDF.