Generalizing from simulation
Start with a robot task
Read how a simulated robot policy can use vision and feedback, then describe its task and test conditions.
Start with: Generalizing from simulation
RedLectern topic
Robot learning
Learning robot behavior through demonstrations · rewards · perception · physical feedback.
For: Technical learners · robotics builders.
After this page you can: Describe a robot-learning task and its evaluation constraints.
Page reviewed
Part of: Artificial Intelligence › Embodied AI
About Robot learning
Robot learning studies how a robot can acquire behavior from examples, interaction, or both. In imitation learning, demonstrations provide examples of behavior to learn; in reinforcement learning, a reward signal guides the agent toward desired outcomes. A robot’s observations may include camera images or other sensor data, while its actions change the physical environment and shape what it observes next.
A useful way to describe a task is to name the goal, observations, actions, learning signal, and conditions for success. For example, an object-pushing task might ask an arm to move an object to a target using camera feedback, while success depends on reaching the target from varied starting positions. Evaluation should also state whether trials are simulated or physical, which objects and starting conditions are held out, what failures or safety constraints count, and how performance is measured. A policy that works in simulation may fail on a physical robot when their dynamics differ, so transfer needs its own evaluation.
Start here
GTI: Learning to Generalize across Long-Horizon Tasks from Human Demonstrations
Learn from demonstrations
See how demonstrations can teach robot behavior and why the data quality and coverage matter.
Start with: GTI: Learning to Generalize across Long-Horizon Tasks from Human Demonstrations
Part 1: Key Concepts in RL
Connect robot learning to reinforcement learning
Review trajectories, rewards, and returns before examining how they enter a robot task.
Start with: Part 1: Key Concepts in RL
Learn the foundations
1.Learning behavior and transferring itA robot can learn from demonstrations that pair situations with actions, or from trial and error guided by a reward.
A robot can learn from demonstrations that pair situations with actions, or from trial and error guided by a reward. Demonstrations can give a policy examples of how to act without requiring random exploration on the physical robot, but the learned behavior may not cover situations missing from the examples. When the robot visits states unlike those in its demonstrations, its actions can drift away from the demonstrated behavior.
Simulation offers many training interactions and can reduce some safety concerns during training, but simulation behavior may depend on simulator characteristics. A sim-to-real evaluation should specify what was randomized or calibrated, what is different in the physical setup, and which tasks or starting conditions were not used for training. Test the transferred policy on the robot and report task success and relevant failures; success in simulation alone does not establish physical performance.
- Sim-to-real transfer of robotic control with dynamics randomization
An example of training in simulation and deploying a policy on a physical robot.
Imitation learning
Learning behavior from demonstrations.
Sim-to-real transfer
Applying learning from simulation to a physical environment.
Try it
Describe an object-pushing task learned in simulation. Name one change you would make to test whether the policy transfers to a physical robot.
The task is to push an object to a target from varied starting positions. A transfer test could use physical trials with held-out starting positions and report how often the object reaches the target.
2.Choose a learning signalA reward turns task goals into a signal that a reinforcement-learning agent can accumulate over a trajectory.
A reward turns task goals into a signal that a reinforcement-learning agent can accumulate over a trajectory. The reward may depend on the current state, action, and next state. For a robot, a task designer might reward reaching a target, but a reward that only recognizes final success gives little direct guidance about intermediate actions. Shaping intermediate rewards can offer denser feedback, while also making it important to check that the agent is optimizing the intended task.
Physical feedback also matters: a robot observes the result of its actions and uses those observations to choose what to do next. A task description should separate the learning signal from the evaluation measure. For example, a reward may encourage progress toward a target, while evaluation records successful completion, collisions, or other task-specific constraints. State which observations are available to the controller and which feedback is used only to evaluate the outcome.
- Generalizing from simulation
Connects a robot's vision observations and physical actions to feedback and reward design.
Try it
For a robot that must push a block to a marked region, propose one reward signal and two separate evaluation measures.
A reward could increase as the block gets closer to the marked region. Evaluation measures could be the fraction of trials where the block ends inside the region and the number of collisions per trial.
3.Perception and evaluation constraintsA robot policy acts on observations, not on an abstract task description.
A robot policy acts on observations, not on an abstract task description. In a vision-based task, the observation may be an image; in other setups it can include depth or sensor feedback. Closed-loop behavior means that the robot uses new observations as it acts, so evaluation should include changes in object placement or appearance that test whether the policy can respond to feedback.
Physical trials can be slow, labor-intensive, unsafe at scale, and hard to reproduce. Simulated evaluation can make controlled variation easier, but it does not replace checking performance on hardware when the intended task is physical. Define success before testing, describe the robot and test conditions, and report task-relevant failures alongside success. Keep comparisons within the same evaluation conditions, including task setup, objects, initial states, and whether human judgment is used.
- Generalizing from simulation
An example of a vision-to-action robot policy using visual feedback.
Try it
Write a one-sentence evaluation plan for a simulated block-pushing policy that you intend to use on a physical arm. Include a held-out condition and one failure to record.
Test the policy on the physical arm with object positions excluded from training, report the fraction reaching the target, and record collisions as failures.
Latest
No recent updates are available.
Books
Reinforcement Learning: An Introduction
Richard S. Sutton and Andrew G. Barto · 2018
Builds the foundations for learning behavior through reward and interaction. A Johns Hopkins syllabus links the second edition online.
Robot Programming by Demonstration: A Probabilistic Approach
Sylvain Calinon · 2009
Takes a practitioner’s perspective on transferring skills to robots by imitation, with examples connected to robotics and machine learning.
Robot Learning from Human Teachers
Sonia Chernova and Andrea L. Thomaz · 2014
Introduces learning task policies from human examples and examines the technical challenges of learning from non-expert teachers.
