Imitation learning from demonstrations

For many manipulation tasks, it is far easier to show a robot what to do than to write a reward function or a controller. Folding a shirt, loading a dishwasher, or plugging in a cable involve subtle sequences of contacts and adjustments that a human operator performs naturally. Imitation learning trains policies from demonstrations of the task, and since around 2023 it has become the leading approach for learning dexterous manipulation, driven by low-cost teleoperation hardware and new policy architectures that handle the variety in human demonstrations. Imitation learning has its own pitfalls, most famously the compounding of small errors once the robot drifts away from the states it saw in training. This lesson covers how demonstrations are collected, behavior cloning, distribution shift and DAgger, the problem of multimodal demonstrations and the architectures that address it (action chunking and diffusion policies), the importance of data scale, diversity and quality, other forms of learning from humans, and how to evaluate learned policies honestly.

Collecting demonstrations

Demonstrations pair observations (camera images, joint states) with the actions the demonstrator took. Common collection methods include:

• Kinesthetic teaching: physically guiding a compliant robot arm through the task while it records joint trajectories. Simple, but the demonstrator's hands appear in camera images and two-handed tasks are awkward.
• Teleoperation with leader-follower arms: the operator moves a small "leader" arm whose joint angles are copied by the robot "follower" arms. Low-cost bimanual systems of this kind have made high-quality data collection accessible to many labs.
• Virtual reality and motion capture: the operator's hand and head motions, tracked by a VR headset or gloves, are mapped to the robot's arms and grippers, useful for humanoids and dexterous hands.
• Handheld data collection devices: hand-held grippers with cameras that record demonstrations without a robot present, later transferred to robots with matching grippers, allowing data collection in many real environments.

Teleoperation quality matters: lag, poor depth perception and missing force feedback make demonstrations slower and less smooth, and policies learn whatever is in the data, including hesitations and mistakes.

Behavior cloning

The simplest approach, behavior cloning (BC), treats imitation as supervised learning: train a network to predict the demonstrator's action from the observation, minimizing the difference between predicted and demonstrated actions. BC is straightforward, uses standard deep learning tools, and can learn complex visuomotor skills from tens to hundreds of demonstrations for a single task. It underlies most modern manipulation policies, including large multi-task models (Lesson 11).

Distribution shift and compounding errors

BC has a fundamental weakness. The policy is trained only on states the expert visited. When it makes a small mistake at test time, it reaches a state slightly outside the training data, where its predictions are less reliable, leading to larger mistakes and states further from the data. Errors compound, and in theory can grow quadratically with the length of the task. The example below shows the effect with a simple lookup-table policy for lane keeping.

import numpy as np
# Behavior cloning versus DAgger for lane keeping with a lookup-table policy
rng = np.random.default_rng(10)
dt = 0.1
def dynamics(s, u):
    y, th = s                                   # lateral offset (m), heading (rad)
    th = th + u * dt + rng.normal(0, 0.02)      # steering changes heading, with noise
    y = y + 2.0 * np.sin(th) * dt               # 2 m/s forward speed
    return np.array([y, th])

def expert(s):
    return float(np.clip(-1.5 * s[0] - 3.0 * s[1], -1, 1))

edges_y, edges_t = np.linspace(-2, 2, 41), np.linspace(-1, 1, 41)
def key(s):
    return (int(np.digitize(s[0], edges_y)), int(np.digitize(s[1], edges_t)))

def rollout(policy, steps=150, start_sd=0.05, push=False):
    s = rng.normal(0, start_sd, 2); visited = []
    for t in range(steps):
        if push and t == 30:
            s = s + np.array([0.0, 0.35])       # a disturbance, like a gust or a pothole
        visited.append(s.copy())
        s = dynamics(s, policy(s))
        if abs(s[0]) > 1.5:
            return visited, True                # left the lane
    return visited, False

def make_policy(table):
    return lambda s: table.get(key(s), 0.0)     # unseen states: go straight

table = {}
for _ in range(50):                             # behavior cloning: expert demonstrations only
    states, _ = rollout(expert)
    for s in states:
        table[key(s)] = expert(s)

def failure_rate(policy, trials=300):
    return np.mean([rollout(policy, push=True)[1] for _ in range(trials)])

print(f"expert failure rate with disturbance: {failure_rate(expert):.1%}")
print(f"behavior cloning ({len(table)} table entries): {failure_rate(make_policy(table)):.1%}")
for it in range(1, 6):                          # DAgger: run the learner, ask the expert to label what it visits
    for _ in range(30):
        states, _ = rollout(make_policy(table), push=True)
        for s in states:
            table[key(s)] = expert(s)
    print(f"DAgger iteration {it} ({len(table)} entries): {failure_rate(make_policy(table)):.1%}")

expert failure rate with disturbance: 0.0%
behavior cloning (19 table entries): 99.7%
DAgger iteration 1 (160 entries): 0.3%
DAgger iteration 2 (162 entries): 0.0%
DAgger iteration 3 (163 entries): 0.0%
DAgger iteration 4 (164 entries): 0.0%
DAgger iteration 5 (164 entries): 0.0%

Trained only on the expert's smooth demonstrations, the behavior cloning policy has seen only a narrow band of states near the lane center. After a disturbance pushes the vehicle into unfamiliar states, it has no idea what to do and leaves the lane almost every time. DAgger (dataset aggregation) fixes this by running the learner's own policy, recording the states it actually visits, asking the expert what it would have done in each, and adding those labels to the training set. After a single iteration, the table covers the states the learner gets itself into, and failures almost disappear.

In real robotics, DAgger-style corrections are gathered by letting the robot run while an operator intervenes when it goes wrong (interactive or human-gated imitation), and by deliberately collecting recovery demonstrations from perturbed states. Adding noise to the demonstrator's actions during collection (so the data includes small deviations and corrections) also helps. Real policies use neural networks that generalize better than a lookup table, but the underlying problem of distribution shift remains.

Multimodal demonstrations

Humans do not perform a task the same way every time. To go around an obstacle, a demonstrator may pass left in some demonstrations and right in others. A policy trained with a simple regression loss predicts the average of these actions, straight into the obstacle. Demonstrations are also noisy and inconsistent in timing. Modern imitation learning architectures address these problems:

• Action chunking: predicting a sequence of future actions (for example the next 50 to 100 steps) rather than one step at a time, executed with temporal smoothing. Chunking reduces compounding errors, handles pauses and hesitations in the data, and produces smoother motion. The Action Chunking with Transformers (ACT) architecture popularized this for bimanual manipulation.
• Diffusion policies: generating action sequences by iteratively denoising random noise, conditioned on observations, just as image diffusion models generate pictures. They represent multimodal action distributions naturally, choosing one coherent way of doing the task rather than averaging several, and have become a leading approach for visuomotor policies. Flow matching provides a related, often faster, generative approach.
• Discretized or tokenized actions, predicted with classification or by transformers as sequences of tokens, which also represent multiple modes.

Data scale, diversity and quality

Policies trained on a few hundred demonstrations in one setting often fail when lighting, backgrounds, object positions or object types change. Generalization requires diverse data: many objects, scenes, lighting conditions, camera positions and operators. Large multi-institution datasets pooling demonstrations from many robots and tasks, such as the Open X-Embodiment collection, support cross-embodiment training (Lesson 11). Data quality matters as much as quantity: consistent, efficient, successful demonstrations produce better policies than a large volume of hesitant or inconsistent ones, and data curation, filtering failed or poor demonstrations, is an important practical step. Collection costs real time and money, so teams plan data collection like an engineering project.

Other ways of learning from humans

• Inverse reinforcement learning infers a reward function that explains the demonstrations, then optimizes it, potentially generalizing beyond the demonstrated states.
• Learning from human videos: internet and egocentric videos of people doing tasks contain vast information about how objects are manipulated, though they lack robot actions; methods extract hand trajectories, object motions or learned latent actions, and use video for pre-training representations.
• Corrections and preferences: humans correct robot behavior during execution or rank alternative behaviors, providing feedback cheaper than full demonstrations.
• Combining imitation and RL: imitation provides a starting policy and RL fine-tunes it in simulation or the real world for robustness and speed.

Evaluating learned policies

Evaluating robot policies is harder than evaluating classifiers. Success depends on random object placements, lighting and small details, so results from a handful of trials are noisy. Good practice includes running many trials (tens to hundreds) with randomized but documented conditions, reporting success rates with confidence intervals, testing on unseen objects and scenes to measure generalization, recording failure modes, and comparing against baselines under identical conditions. Simulation benchmarks help reproducibility but may not reflect real-world performance, so real-world evaluations remain essential for claims about deployment readiness.

Imitation learning in industry

Imitation learning is moving from research labs into commercial applications. Companies use teleoperated demonstrations to teach robots tasks such as kitting, packing irregular items, laboratory sample handling and garment handling, where traditional programming is impractical. A common pattern is a human-in-the-loop service: robots operate autonomously most of the time, remote operators take over when the robot is unsure or fails, and those interventions become new training data. This approach delivers value while the policy is still imperfect and steadily reduces the need for human help, but it requires careful attention to latency, operator training, data privacy and the cost of remote support.

Whatever the setting, the quality of an imitation-learned policy is bounded by the quality of the demonstrations and the coverage of situations it will meet. Planning data collection deliberately, with recoveries, variety and honest evaluation, is the core engineering skill of this approach.

Teams should also budget for re-collecting data when hardware changes. A new gripper, a moved camera or a different robot arm can invalidate a dataset, so documenting the exact setup used for each demonstration is essential for reproducing and extending results.