Prediction and behavior planning

Perception tells an automated vehicle what is around it now. To drive well it must also anticipate what those road users will do next and decide what it should do in response: keep the lane or change, yield or go, nudge around a parked truck or wait. This lesson covers the prediction of other agents' motion and intent, and the behavior planning layer that chooses a maneuver. These two problems are tightly linked, because what others do depends on what the automated vehicle does, and the reverse. We look at physics-based and learned prediction, the idea of multimodal futures, interaction and game-theoretic thinking, and the long-running debate between rule-based and learned decision-making.

Why prediction is hard

A pedestrian standing at the curb may cross, wait or turn and walk along the sidewalk. A car in the next lane may hold its lane or cut in. A cyclist may swerve to avoid a pothole. These futures are uncertain and often multimodal: there are several distinct possibilities rather than one most likely path with small scatter. Averaging them produces nonsense, such as a predicted path that drives halfway between going straight and turning left. A good predictor outputs several possible trajectories with probabilities, and the planner must make a choice that is safe against the plausible ones, not just the most likely.

Prediction horizons are typically 3 to 8 seconds. Over the first second, physics dominates: a vehicle cannot change its velocity much. Beyond a few seconds, intent dominates: where the driver wants to go, what lane they are in, whether they have signaled, and how traffic around them is moving.

Physics-based prediction

The simplest predictors extrapolate current motion. A constant-velocity model assumes the agent continues at its present speed and heading; a constant-turn-rate-and-velocity model, CTRV, also continues its current turning. These work surprisingly well for one or two seconds and are a valuable baseline and fallback, but fail when agents follow the road, stop at signals or react to others.

# Constant velocity vs constant turn rate prediction errors for a turning car
import numpy as np
rng = np.random.default_rng(3)
v, w, dt = 10.0, 0.2, 0.1
t = np.arange(0, 3.01, dt)
# truth: car turns at 0.2 rad/s with slight speed noise
x_true = (v / w) * np.sin(w * t)
y_true = -(v / w) * (np.cos(w * t) - 1)
x_cv, y_cv = v * t, np.zeros_like(t)
w_est = w + rng.normal(0, 0.02)              # noisy yaw-rate estimate
x_ct = (v / w_est) * np.sin(w_est * t)
y_ct = -(v / w_est) * (np.cos(w_est * t) - 1)
for T in (1.0, 2.0, 3.0):
    k = int(round(T / dt))
    e_cv = np.hypot(x_cv[k] - x_true[k], y_cv[k] - y_true[k])
    e_ct = np.hypot(x_ct[k] - x_true[k], y_ct[k] - y_true[k])
    print(f"t={T:.0f} s  CV error {e_cv:5.2f} m   CTRV error {e_ct:5.2f} m")

t=1 s  CV error  1.00 m   CTRV error  0.20 m
t=2 s  CV error  3.98 m   CTRV error  0.81 m
t=3 s  CV error  8.91 m   CTRV error  1.81 m

Map-aware and learned prediction

Vehicles mostly follow lanes. Map-aware predictors use the HD map from Lesson 3 to enumerate the lanes an agent could take, such as continuing straight, turning right or changing lanes, and then estimate the probability of each based on position, speed, heading, turn signals and history. Each hypothesis becomes a trajectory along its lane path with a speed profile. This approach is interpretable and handles common cases well.

Learned predictors, trained on millions of recorded interactions, take as input the recent history of all nearby agents and a representation of the map, often as a graph of lane segments or a rasterized bird's-eye image, and output several trajectories with probabilities. Modern architectures use transformers that attend jointly over agents and map elements. They capture subtle cues, such as a pedestrian's body orientation or a car slowing slightly before a turn, that are hard to code by hand. They are evaluated on metrics such as minimum average displacement error over the top k predictions, minADE, and miss rate, the fraction of cases where no prediction comes within a threshold of the true future. Public benchmarks built on large driving datasets have driven much of this progress.

Interaction and game-theoretic thinking

Driving is a negotiation. At a four-way stop, at a merge, or when a pedestrian hesitates at a crosswalk, each party's action depends on what it expects the others to do. If the automated vehicle predicts others as if they ignore it, it may freeze when it could proceed, because every gap looks blocked, or it may assume others will yield when they will not. Game theory frames such interactions as players choosing actions to optimize their own payoffs, given beliefs about others. Full game-theoretic solutions are computationally expensive, but simplified ideas are widely used: conditional prediction, which asks how others would respond if the vehicle took a given action; leader-follower models, where one party commits and the other reacts; and courtesy terms that discourage actions forcing others into hard braking.

A notorious failure mode is the frozen robot: a vehicle so cautious that it never finds a safe gap in dense traffic and blocks the road. Human drivers resolve this by creeping forward to signal intent, which changes others' behavior. Automated vehicles increasingly do the same, using small, legible motions to communicate. Legibility, behaving so that others can easily predict you, is itself a safety property.

Behavior planning

The behavior planner chooses the high-level maneuver: follow the lane, change lanes left or right, prepare to turn, stop at the line, yield to a pedestrian, pull over. Its output constrains the motion planner of Lesson 5, which computes the exact trajectory. Classic designs use finite-state machines: states such as lane follow, prepare lane change, change lane and stop, with transitions guarded by conditions like gap available or signal red. State machines are transparent and easy to test, but the number of states and transitions explodes as the operational design domain grows, and edge cases fall between rules.

Other designs score candidate behaviors with a cost function that weighs safety, progress, comfort and rule compliance, choosing the lowest-cost option. Hierarchical behavior trees, borrowed from robotics and games, organize decisions into reusable subtrees. Probabilistic approaches such as partially observable Markov decision processes, POMDPs, explicitly reason about uncertain intent: is that car going to turn, and how will the answer change as I learn more? They are principled but computationally heavy, so practical systems approximate them.

Rule-based versus learned decision-making

A major architectural debate concerns how much of the decision-making should be learned. At one end, modular stacks with hand-engineered rules make each decision traceable to a requirement, which suits safety cases and regulation. At the other, end-to-end systems train a single network from sensor input to trajectory, learning from large quantities of human driving and simulation. Proponents argue end-to-end learning scales better to the long tail of situations that rules never anticipate; critics point out that such systems are harder to verify, may copy human errors and can fail unpredictably outside their training distribution.

Most deployed systems at the time of writing (October 2026) blend the two. Learned components handle perception and prediction and propose candidate trajectories, while rule-based layers enforce hard constraints: never enter an occupied crosswalk, never exceed the speed limit by more than a set margin, always keep a safe following distance. These guardrails can be checked independently, an idea related to the responsibility-sensitive safety model discussed in Lesson 5. Whatever the architecture, the system must be able to explain, at least after the fact, why it did what it did, so that incidents can be investigated.