Why robots learn

Every robot in the earlier courses of this program did what an engineer told it to do. ROB-300 gave you controllers derived from equations of motion, ROB-340 gave you maps built by probabilistic filters, ROB-350 gave you planners that search configuration space, and ROB-400 gave you grasp analysis built on friction cones and grasp matrices. Those methods are powerful, well understood and, in many factories, entirely sufficient. This course is about the cases where they are not: where the world is too varied, the physics too uncertain or the task too hard to specify for an engineer to write the solution down directly. In those cases, robots increasingly learn part of their behaviour from data, either from their own trial and error or from demonstrations by people.

This first lesson sets the scene. It explains what robot learning is, where it helps and where it does not, how the main families of methods relate to one another, and why learning on a physical robot is so much harder than learning to play a video game or to recognize photographs. It also introduces the vocabulary and the habits of measurement that the rest of the course relies on. Readers who have taken CMPUT-367 will already know supervised learning, loss functions and gradient descent; we use those ideas freely and point back to that course rather than re-teaching them.

What robot learning means

In this course, robot learning means improving a robot's performance on a task by using data rather than by hand-writing every rule. The data can come from many places: the robot's own attempts, logged by its sensors; demonstrations by a person driving the robot through teleoperation or moving it by hand; simulation, where millions of attempts are cheap; large internet data sets of images, video and text; and logs from fleets of robots already deployed. The thing being learned can also vary. A robot might learn a whole policy, which maps observations directly to actions. It might learn a single component, such as a grasp-quality predictor, a perception model that segments objects, or a dynamics model that predicts what happens when it pushes something. Or it might learn a reward or cost function that a classical planner then optimizes.

Three broad families of methods recur throughout the course. Reinforcement learning, or RL, improves a policy by trial and error, guided by a scalar reward signal that says how well each attempt went. Imitation learning copies the behaviour of an expert, usually a human teleoperator, from recorded demonstrations. Representation and foundation-model learning trains large models on broad data, often not collected by robots at all, and then adapts them to robot tasks. Real systems combine these: a vision encoder pretrained on internet images, a policy initialized from human demonstrations, and a final round of reinforcement learning in simulation to make it robust.

Why not just engineer everything?

Hand-engineered robotics has a strong track record. A car body shop with hundreds of welding robots runs almost entirely on explicitly programmed trajectories, and it achieves reliability that learned systems still struggle to match. So the first question to ask about any learning project is whether learning is actually needed. It usually is when one or more of the following holds.

The task is hard to specify. You can write down the dynamics of a two-link arm, but writing down a rule for folding a crumpled T-shirt, for wiping a spill of unknown shape or for sorting mixed recycling is far harder. In such tasks, people find it much easier to show what they want than to describe it.

The world is too varied. A warehouse robot may meet hundreds of thousands of distinct products. A home robot faces an effectively unbounded variety of objects, lighting conditions and layouts. Hand-tuned heuristics tend to work on the cases the engineer imagined and fail on the long tail of cases they did not.

The physics is uncertain or expensive to model. Contact with deformable objects, granular materials, liquids or rough terrain involves physics that is either unknown or too slow to simulate accurately in real time. A legged robot crossing loose gravel does not have a clean analytical model to plan with.

Perception is the bottleneck. Recognizing objects, estimating their pose and judging where they can be grasped from raw camera images is a problem where learned models are now far better than hand-designed feature pipelines, as ROB-330 showed.

Conversely, learning is usually the wrong tool when a clean model exists and the environment is controlled, when every failure is very costly and must be explained, when there is little data and no good simulator, or when a certification regime requires deterministic, verifiable behaviour. Many of the most successful deployed systems are hybrids: learned perception feeding a classical planner and controller with well-understood safety properties.

What makes robot learning hard

The headline successes of machine learning, such as image classifiers, language models and game-playing agents, rely on enormous amounts of data that are cheap to collect or generate. A board game can be played millions of times a day in software. Physical robots break that pattern in several ways, and understanding them explains many of the design choices you will meet later.

Data is expensive. A robot arm attempting a grasp takes a few seconds per try, needs an operator to reset the scene, wears out its gripper and sometimes breaks the object. A large research data set of real robot trajectories may contain on the order of a million episodes collected across many labs and robots, which sounds large but is tiny compared with the billions of images or trillions of words used to train vision and language models.

Mistakes have physical consequences. A learning agent explores by trying things that might not work. In a game that costs nothing; on a robot it can mean collisions, dropped objects, damaged hardware or, near people, injury. Exploration must therefore be constrained, supervised or moved into simulation.

Embodiment differs. A policy learned on one robot arm, with its particular kinematics, cameras and gripper, often does not transfer to a different arm. Even the same model of robot can behave differently because of wear, calibration and lighting. This makes it hard to pool data across platforms, although recent cross-embodiment data sets aim to do exactly that.

Observations are partial and noisy. Cameras see surfaces, not mass, friction or hidden objects, and the robot's own arm often blocks the view at the critical moment. Learned policies must act well despite uncertainty about the true state, a theme formalized in lesson 2.

Rewards are hard to define. Telling a robot that it succeeded at clearing a table, folding a shirt or assembling a part often requires a human judgement or a carefully engineered detector. Poorly designed rewards produce surprising and unwanted behaviour, as lesson 6 discusses.

Evaluation is slow and noisy. To compare two policies, you must run each many times, in conditions that are as similar as possible. Because each trial is costly, many published comparisons rest on a few dozen trials, and their confidence intervals are wide. Learning to report and read those intervals honestly is one of the most useful skills this course teaches.

# How long does it take to collect robot data compared with simulation?
seconds_per_episode = 20      # real robot: attempt plus scene reset
hours_per_day = 8             # supervised operation
robots = 4
real_per_day = robots * hours_per_day * 3600 / seconds_per_episode
print(f"real episodes per day: {real_per_day:,.0f}")
for target in (1_000, 50_000, 1_000_000):
    print(f"{target:>9,} episodes: {target / real_per_day:8.1f} days on {robots} robots")
# A simulator stepping 4,096 environments in parallel at 1,000 steps/s each
envs, steps_per_s, steps_per_episode = 4096, 1000, 200
sim_per_hour = envs * steps_per_s * 3600 / steps_per_episode
print(f"simulated episodes per hour: {sim_per_hour:,.0f}")

real episodes per day: 5,760
    1,000 episodes:      0.2 days on 4 robots
   50,000 episodes:      8.7 days on 4 robots
1,000,000 episodes:    173.6 days on 4 robots
simulated episodes per hour: 73,728,000

Learning in the loop: where learned components sit

It is useful to picture a robot system as a loop: sensors produce observations, a perception stage turns them into a state estimate, a decision stage chooses actions, a control stage turns those actions into motor commands, and the world responds. Learning can be inserted at almost any point in that loop, and each choice has different consequences for data needs, safety and debugging.

Modular designs learn one stage and engineer the others. A learned object detector might feed a grasp planner from ROB-400 and a motion planner from ROB-350. The advantage is that each module can be tested, swapped and reasoned about separately, and the safety-critical low-level control stays engineered. The disadvantage is that errors at module boundaries compound, and the interfaces between modules may throw away information that later stages needed.

End-to-end designs learn a single policy that maps raw observations, such as camera images and joint angles, directly to actions, such as joint velocities or end-effector poses. They can, in principle, use all the information in the observations and avoid hand-designed interfaces. They are also harder to debug, need more data and are harder to certify. Most of the recent large vision-language-action models discussed in lesson 11 are end-to-end at the decision level but still rely on engineered low-level controllers underneath.

Residual and hybrid designs keep a classical controller and learn a correction on top of it. For example, a nominal impedance controller from ROB-300 might be augmented by a learned residual that compensates for unmodelled friction during insertion. This often gives most of the benefit of learning with much less data, because the classical part already handles the easy cases.

Measuring success honestly

A learning project lives or dies on its measurements. Before training anything, write down how success will be measured, on which tasks, objects and conditions, and how many trials will be run. Decide in advance what counts as a success, such as an object lifted 10 cm and held for 3 s, so that you cannot redefine success after seeing the results. Keep a held-out set of objects or scenes that are never used during training, because a policy that has memorized its training objects tells you little about how it will do on new ones.

Success rates are proportions estimated from a limited number of trials, so they carry uncertainty. If a policy succeeds on 18 of 20 trials, its true success rate is not known to be 90 %; a 95 % confidence interval runs from roughly 70 % to 97 %. Lesson 12 introduces the Wilson interval, which behaves well for small samples and for rates near 0 or 1, but even the simple normal approximation shows how wide these intervals are.

Here \( \hat{p} \) is the observed success fraction and \( n \) is the number of trials. With \( \hat{p} = 0.9 \) and \( n = 20 \), the half-width is \( 1.96\sqrt{0.09/20} \approx 0.13 \), so the simple interval is roughly 77 % to 100 %, which is already wide enough to make many claimed improvements meaningless. The normal approximation also misbehaves near 100 %, which is one reason lesson 12 prefers the Wilson interval.

A short history

Learning has been part of robotics for decades, but its role has changed sharply. Early work in the 1980s and 1990s used reinforcement learning and neural networks on small problems such as balancing a pole on a cart and learning simple motor skills, and learning from demonstration, sometimes called programming by demonstration, let operators teach industrial robots by guiding them. In the 2000s, policy search methods learned helicopter aerobatics and robot table tennis from a mix of demonstrations and trial and error.

The deep learning wave of the 2010s changed perception first. Convolutional networks made object detection and grasp prediction from images practical, and large data-collection efforts showed that robots could learn grasping from hundreds of thousands of their own attempts. Deep reinforcement learning, popularized by agents that learned to play video games from pixels, was applied to simulated and then real robots. Simulation with heavy randomization allowed learned controllers, such as legged locomotion policies, to transfer to real machines.

In the 2020s, three trends converged. Low-cost teleoperation hardware made it practical to collect thousands of high-quality human demonstrations for dexterous tasks. Generative models, such as diffusion models and transformers, proved very good at representing the varied ways people perform a task. And large models pretrained on internet images and text were adapted to output robot actions, giving rise to the vision-language-action models of lesson 11. At the time of writing (October 2026), these models are advancing quickly, and claims about their generality need careful, sceptical evaluation, which is why this course spends so much time on measurement.

How this course is organized

Lessons 2 to 6 build reinforcement learning from the ground up: Markov decision processes, dynamic programming and its link to optimal control, value-based and policy-gradient methods, and the problem of designing rewards that produce safe, useful behaviour. Lesson 7 covers simulation as the main source of cheap experience. Lessons 8 and 9 cover imitation learning, from simple behaviour cloning to modern generative policies and the teleoperation systems used to collect demonstrations. Lesson 10 covers how robots learn useful representations of what they see. Lesson 11 surveys foundation models for robots. Lesson 12 brings everything together into a deployable, measurable learning pipeline for a pick-and-place skill.

Throughout, code examples use small, self-contained NumPy programs with fixed random seeds, so you can run them and get the same numbers. Listings that depend on robot hardware or large frameworks are marked as not run. The goal is not to make you an expert user of a particular library, which will change, but to give you a clear understanding of the ideas that every library implements.