Simulation for robot learning

Lesson 5 showed that a modern on-policy algorithm can consume a hundred million environment steps before a locomotion policy works well. No physical robot can supply that. Simulation can: a single workstation with a modern graphics processor can step thousands of simulated robots in parallel, faster than real time, with no wear, no resets by people and no risk of damage. Simulation has therefore become the main source of experience for reinforcement learning in robotics, and a growing source of synthetic data for imitation and perception. The catch is that a policy trained in simulation must still work on a real robot. This lesson covers how robot-learning simulators work, how to make policies robust to the differences between simulation and reality, and how curricula and parallelism make training practical. ROB-360 covered simulation fundamentals, physics engines, models and the basics of sim-to-real transfer; here we focus on what is specific to learning.

What a learning simulator needs

A simulator for robot learning must do four things well. It must be fast, because learning needs enormous numbers of steps; throughput, measured in total environment steps per second across all parallel copies, matters more than the speed of any single copy. It must be physically plausible for the contacts that matter to the task, such as feet on terrain or fingertips on objects. It must produce the observations the real robot will see, which may include joint encoders, force sensors, depth images or color images. And it must be controllable from code, so that every parameter, from object masses to lighting, can be changed at each reset.

At the time of writing (October 2026), several open-source and commercial platforms meet these needs to different degrees. GPU-based simulators such as NVIDIA's Isaac Sim and its Isaac Lab learning framework run thousands of environments on one graphics card. MuJoCo, open-sourced in 2022 and now with GPU-accelerated variants, is widely used for contact-rich control research. PyBullet, Gazebo and Webots remain common for smaller experiments and ROS-based workflows. Libraries change quickly; the concepts in this lesson apply to all of them.

The reality gap

A policy trained in simulation inherits every assumption the simulator makes. Real motors have backlash, friction that changes with temperature, current limits and communication delays. Real contacts are softer or stiffer than modeled, and friction coefficients vary from surface to surface. Real cameras have noise, motion blur, lens distortion and changing lighting. Real objects have masses and inertias that differ from their nominal values. The difference between simulated and real behavior is called the reality gap, or the sim-to-real gap.

Learning makes the gap more dangerous than it is for hand-designed controllers. An optimizer searching over millions of trials will find and rely on any quirk of the simulator that helps it score: a contact model that allows a foot to slide without resistance, a gripper that never slips, a joint that responds instantly. Such a policy can look excellent in simulation and fail immediately on hardware. The tools below aim to train policies that do not depend on those quirks.

Domain randomization

Domain randomization trains the policy across many variations of the simulator, so that the real world looks like just one more variation. At each episode reset, physical parameters are drawn from ranges: masses, friction coefficients, motor strength, joint damping, sensor noise, control latency, and for vision policies also textures, colors, lighting, camera pose and distracting objects. A policy that succeeds across the whole range cannot depend on any single value, so it is more likely to work at the real values, which are never known exactly.

Choosing ranges is a balancing act. Ranges that are too narrow may not include reality. Ranges that are too wide make the task harder than necessary and can produce timid, overly conservative policies that perform poorly everywhere. A good starting point is to measure the real parameters as well as possible, as ROB-360's system identification lesson described, and then randomize around those measurements with ranges that reflect the remaining uncertainty, plus some margin. Randomizing latency and action delays is particularly important for dynamic tasks, because many sim-to-real failures come from timing rather than from physics.

# Why randomization helps: tune a push force on a nominal sim vs a randomized sim
import numpy as np
rng = np.random.default_rng(3)
def slides_to_target(force, mu, m=1.0, push_time=0.3, g=9.81, target=(0.45, 0.55)):
    # Push a block with constant force for push_time, then let friction stop it.
    a_push = force / m - mu * g
    if a_push <= 0:
        return False
    v = a_push * push_time
    d = 0.5 * a_push * push_time**2 + v**2 / (2 * mu * g)
    return target[0] <= d <= target[1]
forces = np.linspace(3, 12, 181)
mu_nominal = 0.30
best_nominal = max(forces, key=lambda f: slides_to_target(f, mu_nominal) - abs(f - 6.0) * 1e-6)
mus_train = rng.uniform(0.22, 0.40, 400)            # randomized friction in training
rate = [np.mean([slides_to_target(f, mu) for mu in mus_train]) for f in forces]
best_random = forces[int(np.argmax(rate))]
mus_real = rng.normal(0.33, 0.03, 2000)             # "reality": unknown, slightly different
for name, f in (("nominal-tuned", best_nominal), ("randomized-tuned", best_random)):
    real = np.mean([slides_to_target(f, mu) for mu in mus_real])
    print(f"{name:17s} force {f:5.2f} N -> real success {real:.1%}")

nominal-tuned     force  7.10 N -> real success 14.1%
randomized-tuned  force  7.90 N -> real success 47.9%

Beyond uniform randomization

Several refinements improve on picking fixed ranges by hand. Automatic domain randomization starts with narrow ranges and widens each one whenever the policy performs well at its edges, so the difficulty grows with the policy's ability. Adaptive or system-identification-in-the-loop approaches use real-world data to adjust the simulator's parameter distribution so that simulated trajectories match real ones more closely. Privileged-information training, sometimes called teacher-student training, first trains a teacher policy that can see true simulator parameters such as friction and terrain height, then trains a student policy that sees only the real robot's sensors to imitate the teacher. The student learns to infer hidden parameters from its recent history of observations and actions, which has proved very effective for legged locomotion over rough terrain.

Curricula

Many tasks are too hard to learn from scratch: a quadruped that has not learned to stand will never discover how to climb stairs. A curriculum orders training from easy to hard. It can change the environment, for example starting on flat ground and gradually raising terrain roughness; change the goal, for example starting with targets near the hand and moving them farther away; or change the reward, for example starting with dense shaping and fading it out. Automatic curricula promote each parallel environment to harder settings when its policy succeeds often enough and demote it when it fails, so that every environment trains near the edge of the policy's ability.

Parallel environments in practice

GPU simulation changes how training code is organized. Instead of one environment at a time, the code works with tensors holding the states of all environments at once; every operation, from computing rewards to resetting environments that have finished, is vectorized. Environments finish at different times, so resets are handled per environment within the batch. Observation normalization statistics, curricula and randomization are maintained for the whole batch. Large batches of experience suit on-policy algorithms such as PPO, while off-policy methods benefit less because the bottleneck shifts from collecting data to training on it.

# Isaac Lab / Gymnasium style vectorized loop (listing only, not run here)
import torch
obs, info = env.reset()                         # obs: tensor [num_envs, obs_dim]
for it in range(num_iterations):
    for t in range(steps_per_update):           # collect a rollout from all envs at once
        with torch.no_grad():
            dist = policy(normalizer(obs))
            act = dist.sample()
        next_obs, rew, terminated, truncated, info = env.step(act)
        buffer.add(obs, act, rew, terminated, truncated, dist.log_prob(act))
        obs = next_obs                          # finished envs are reset inside env.step
    ppo.update(buffer)                          # several epochs of mini-batch updates
    curriculum.update(info["episode_progress"])

Validating transfer

Transfer should be tested in stages rather than in one leap. First, evaluate the trained policy in simulation under parameter values outside its training ranges to see how quickly performance degrades. Second, run it in a second simulator or with a more detailed model, which catches dependence on one engine's quirks. Third, test on the real robot at reduced speed and in a protected setting, logging everything. Fourth, compare real and simulated trajectories for the same commands, which often reveals a specific mismatch, such as a slower actuator response or higher friction, that can be fixed by improving the model or widening randomization. Finally, measure real-world success rates with confidence intervals, as lesson 12 describes, before increasing autonomy. ROB-360 covered metrics for quantifying the reality gap, such as comparing distributions of key trajectory features between simulation and reality.