Sensors and perception for automated vehicles

An automated vehicle can only respond to what it perceives. Perception must find every relevant object around the vehicle, from cars and trucks to pedestrians, cyclists, animals and debris, estimate where each one is and how it is moving, and identify the static world: lanes, curbs, traffic signs, signals and drivable space. It must do so in daylight and darkness, in rain, fog and snow, at highway speeds and in crowded downtown streets. ROB-220 described sensors in general and ROB-330 covered computer vision. This lesson looks at the sensor suites used on road vehicles, how their coverage is designed, how their data is fused, and why weather, including a Canadian or northern US winter, remains one of the hardest problems in the field.

The automotive sensor suite

Four sensor types dominate. Cameras are cheap, high-resolution and the only sensor that reads colour, text and signals, but they measure depth only indirectly and suffer in glare and low light. Radar measures range and radial velocity directly using the Doppler effect, works in rain, fog and darkness, and sees through spray, but has coarse angular resolution and can be confused by metal objects such as manhole covers and overhead signs. Lidar fires laser pulses and measures their return time, producing precise three-dimensional point clouds day or night, but it is more expensive, has limited range on dark surfaces and degrades in heavy precipitation. Ultrasonic sensors measure short distances, a few metres, cheaply and are used for parking and low-speed manoeuvres.

Different developers weigh these sensors differently. Most robotaxi and driverless truck programs at the time of writing (October 2026) use all three main modalities, camera, radar and lidar, so that each covers the others' weaknesses. Some driver-assistance and supervised driving systems rely mainly on cameras with neural networks that estimate depth and motion. The trade-off is between cost and simplicity on one side and redundancy and directly measured geometry on the other. For a Level 4 system that must perform its own fallback, regulators and safety assessors usually expect redundancy, so that no single sensor failure leaves the vehicle blind.

Designing coverage

Sensor placement is planned so that the vehicle has overlapping coverage all around, with the longest range where it matters most. A car travelling forward at highway speed needs to see far ahead, perhaps 200 m or more, to detect stopped traffic in time. At an intersection it needs wide-angle views to the sides to see cross traffic. When merging or changing lanes it needs rear and side coverage to detect fast vehicles approaching from behind. When pulling away from the curb it needs near-field coverage to detect a child or a pet next to the bumper. Engineers draw coverage maps showing each sensor's field of view and range, and check that every region relevant to the ODD is covered by at least two sensors of different types.

From raw data to objects

Perception pipelines detect objects in each sensor, then associate and track them over time. Camera networks output two-dimensional or three-dimensional bounding boxes, segmentation masks and classes. Lidar networks operate on point clouds, often after converting them to a bird's-eye-view grid, and output three-dimensional boxes with headings. Radar provides detections with measured radial velocity. A tracker, usually built on the Kalman and multi-target tracking ideas from ROB-310, associates detections across frames, estimates each object's position, velocity, heading and size, and keeps track identities stable. The output is a list of tracked objects with uncertainty, plus static map elements such as lane boundaries and traffic light states.

Many modern stacks fuse sensor data inside the neural network rather than after detection. In a bird's-eye-view, or BEV, architecture, features from all cameras are projected into a common top-down grid around the vehicle, often combined with lidar and radar features, and a single network outputs objects, lanes and occupancy. Occupancy networks go further and predict, for every cell in a three-dimensional grid around the vehicle, whether it is occupied and how it is moving, which helps with unusual objects that do not fit any trained class, such as a fallen ladder or an overturned shopping cart.

Fusing measurements with uncertainty

Whatever the architecture, the principle of fusion is the same as in ROB-310: combine measurements in proportion to how much you trust them. If a radar measures an object's range with a standard deviation of 0.5 m and a camera-based estimate has a standard deviation of 2 m, the fused estimate should lean heavily on the radar. For independent Gaussian measurements of the same quantity, the minimum-variance combination weights each measurement by the inverse of its variance.

# Inverse-variance fusion of radar and camera range estimates (toy example)
import numpy as np
rng = np.random.default_rng(42)
true_range = 48.0
sig_radar, sig_cam = 0.5, 2.0
radar = true_range + rng.normal(0, sig_radar, 1000)
cam = true_range + rng.normal(0, sig_cam, 1000)
w_r, w_c = 1 / sig_radar**2, 1 / sig_cam**2
fused = (w_r * radar + w_c * cam) / (w_r + w_c)
for name, est in (("radar", radar), ("camera", cam), ("fused", fused)):
    print(f"{name:6s} mean {est.mean():6.2f} m  std {est.std():5.2f} m")
print(f"predicted fused std {(w_r + w_c) ** -0.5:5.2f} m")

radar  mean  47.99 m  std  0.49 m
camera mean  47.84 m  std  2.03 m
fused  mean  47.98 m  std  0.48 m
predicted fused std  0.49 m

Weather and the northern winter

Weather affects every sensor. Rain and snow create false lidar returns and scatter laser light, so effective range falls; wet roads reflect lights and blur lane markings for cameras; spray from trucks blinds sensors at highway speed. Fog attenuates lidar and cameras strongly while radar continues to work. Low sun near sunrise and sunset dazzles cameras, a particular problem in winter at northern latitudes when the sun stays low all day. Snow covers lane markings and curbs, changes the shape of the road edge and builds up on sensor windows, while salt spray dries into an opaque film.

Developers respond in several ways. Hardware measures include heated and wiped sensor windows, air knives and washer jets, and placing sensors where they stay cleaner. Software measures include filtering precipitation noise from point clouds, training networks on winter data, relying more on radar and on maps when markings are hidden, and monitoring sensor health so that the system knows when its perception is degraded. Finally, the ODD can exclude conditions the system cannot handle, so that the vehicle reaches a minimal risk condition or does not start a trip when heavy snow is forecast. Canadian research groups, test tracks and university teams have contributed substantially to winter driving datasets and testing, and many operators that serve northern cities treat snow as one of the last conditions to add to the ODD.

Measuring perception performance

Perception is evaluated with metrics such as precision and recall for detection, mean average precision over intersection-over-union thresholds, tracking metrics such as identity switches, and position and velocity errors. For safety, aggregate averages are not enough. What matters is performance on the objects and situations that could cause harm: a pedestrian partly hidden by a parked van, a cyclist at night, a motorcycle between lanes. Developers therefore define safety-relevant metrics, such as the time between an object becoming visible and the first stable track, or the maximum distance at which a child-sized pedestrian is reliably detected, and track them across releases.