Cameras: forward, surround and driver monitoring

Ines Moreau drives for a rideshare company, and in April her hybrid's odometer passed 230,000 km (143,000 miles). She came in for an oil change with a windshield that had taken a stone on a gravel stretch of highway east of Calgary (Denver). The chip sat low on the passenger side, well away from the camera, and a repair shop had filled it. But she had a question for Kiran. "Since the chip, the car tells me the camera is temporarily unavailable some mornings. It always comes back. Is it the chip?" Kiran looked at the windshield from inside the car. The chip was nowhere near the camera's view. The fog on the inside of the glass, right in front of the camera housing, told a different story.

This lesson covers the cameras driver assistance systems use: the forward camera behind the windshield, the wide-angle cameras of surround view and rear vision, the side and mirror-replacement cameras on some newer vehicles, and the infrared cabin cameras that monitor the driver. For each, it covers what the camera sees, what limits it, how it is mounted and what a technician must check.

How a driver assistance camera sees

A driver assistance camera is a small digital camera with a lens, an image sensor and a processor, built to automotive temperature and vibration standards. The image sensor is an array of light-sensitive pixels; many forward cameras use sensors of one to eight megapixels with a very wide dynamic range, so that they can see detail both in bright sky and in a dark underpass in the same frame. The processor, either in the camera itself or in a separate module, runs software that finds lane markings, road edges, vehicles, pedestrians, cyclists, traffic signs and lights in each frame, and tracks them from frame to frame to estimate their position and motion.

A single camera does not measure distance directly. It infers distance from where an object appears in the image, how large it appears, and how that changes over time, all of which depend on knowing the camera's height above the road and the angle at which it points. That is why a forward camera's calibration is so sensitive: if the module believes the camera points slightly up when it actually points level, every object appears closer than it is. Some vehicles use a stereo camera, two cameras a short distance apart, which can measure distance by comparing the two images, much as human eyes do.

Cameras have the same limits as eyes. They need light and a clear view. Low sun shining straight into the lens, heavy rain or snow, fog, a dirty or iced windshield, worn or snow-covered lane markings and the glare of oncoming headlights at night all reduce what the camera can see. When the software's confidence drops, it disables the features that depend on it and tells the driver. A "camera temporarily unavailable" message is often the system working correctly, not a fault.

# What a forward camera can resolve: pixels across a car at different distances
import math
h_fov = 52.0          # horizontal field of view, degrees (illustrative)
px = 1920             # pixels across the image
car_w = 1.8        # width of a car
per_deg = px / h_fov
print(f"about {per_deg:.0f} pixels per degree")
for d in (25, 50, 100, 150):
    ang = math.degrees(2 * math.atan(car_w / 2 / d))
    print(f"car at {d:3} m: {ang:4.2f} deg wide, about {ang * per_deg:4.0f} pixels")

about 37 pixels per degree
car at  25 m: 4.12 deg wide, about  152 pixels
car at  50 m: 2.06 deg wide, about   76 pixels
car at 100 m: 1.03 deg wide, about   38 pixels
car at 150 m: 0.69 deg wide, about   25 pixels

The forward camera and the windshield

The forward camera is usually mounted at the top centre of the windshield, behind the rear-view mirror, in a housing clipped to a bracket that is bonded to the glass. It looks through a specific area of the windshield, and the glass in that area becomes part of the optical system. That has three consequences. The glass must be optically correct: a replacement windshield must match the original in the camera's viewing area, including its thickness, curvature, tint and any coatings, and the bracket must be in exactly the original position. The viewing area must stay clear: many vehicles heat the glass in front of the camera to clear fog and frost, and a housing that is not seated against the glass lets cabin air in to fog the glass inside the housing. And any change to the glass or bracket requires a calibration, because even a small change in the camera's angle or the glass's optics moves the image.

Ines's problem was the second one. Kiran removed the camera cover and found that the light shield, the moulded cone that sits between the lens and the glass, had a cracked clip, so the housing sagged a millimetre or two away from the windshield. Humid air from the cabin, warmed by Ines's heater on cold mornings, condensed on the cold glass inside the housing, and the camera saw fog until the heated area cleared it. The chip had nothing to do with it. A new light shield clip, a clean of the inside of the glass in the camera area, and a check that the camera's bracket had not moved fixed the complaint. Because the camera itself had not been removed from its bracket and the bracket had not moved, the maker's procedure called for a verification road test rather than a full calibration, and the camera's self-calibration status read complete after the drive.

Rock chips and cracks in the camera's view are a different matter. A chip in the viewing area can scatter light and distort the image, and most makers do not allow a repair in that zone; the windshield must be replaced. Windshield replacement on a camera-equipped vehicle is now one of the most common triggers for an ADAS calibration, and the glass, bracket and calibration questions come up again in the collisions and glass lesson.

Surround view, rear and side cameras

Surround view systems use four or more wide-angle cameras, typically one in the grille or front bumper, one under each side mirror and one in the tailgate or rear bumper. Each has a fish-eye lens that sees close to 180 degrees. The module corrects each image's distortion, transforms it to look as if seen from above, and stitches the four images into a single bird's-eye view, with an image of the vehicle in the middle. The stitching depends on the module knowing each camera's exact position and angle, so surround view systems need calibrating when a camera, mirror, bumper or tailgate is replaced, and the calibration usually uses mats or targets laid on the floor around the vehicle.

The rear camera is the oldest of these. Backup cameras have been required on new light vehicles in the US since May 2018 under FMVSS 111, and in Canada under the matching Canadian rule since the same model years, and they provide the rear view for parking guidance lines and, on many vehicles, rear cross-traffic and rear automatic braking. A dirty, misted or water-filled rear camera is one of the most common camera complaints a shop sees.

Some newer vehicles replace exterior mirrors with cameras and in-cabin displays, and many commercial vans use side cameras to show the area beside and behind the vehicle. Hal's trial electric van has a camera on the passenger side that shows the blind zone on a screen whenever the turn signal is on. Those cameras, too, have aiming requirements and, on some vehicles, calibrations after a mirror or door repair.

Driver monitoring cameras

A driver monitoring camera looks at the driver rather than the road. It is usually mounted on the steering column, the instrument panel or near the rear-view mirror, and it uses infrared light-emitting diodes and an infrared-sensitive sensor, so that it works at night and through most sunglasses. The software tracks the driver's head position, eye direction and eyelids, and decides whether the driver is watching the road, distracted or drowsy. Level 2 features that allow hands-off driving depend on it: if the driver looks away for too long, the system warns, then disengages. Many vehicles now use the same camera to detect a drowsy or distracted driver even when no assistance feature is engaged.

Hal's trial van has one, and the drivers noticed it quickly. Marcus Bell, who drives the van most days, came in to ask why it chimed at him so often on the morning route. Kiran checked the van's settings and the camera's view. Marcus wore a cap with a long brim, pulled low against the low winter sun, and the camera lost sight of his eyes whenever he looked down at the delivery tablet mounted below the dash. The camera was working; the tablet mount was in the wrong place. Hal's fleet manager moved the tablet higher, and the chimes stopped. The lesson Kiran wrote in the ADAS log was that cabin cameras are part of the system too, and their complaints often come from the cabin, not the camera.

Inspecting and servicing cameras

A camera inspection starts with the view. The glass or lens in front of the camera must be clean on both sides, free of chips, scratches, film and stickers in the viewing area, and, for a forward camera, free of anything the owner has added: phone mounts, toll transponders, dash cameras and parking permits placed in the camera's view all cause trouble. Many makers mark the area where nothing may be mounted. Next comes the mounting: the bracket bonded to the glass, the camera clipped or bolted into it, the light shield and cover seated, and no sign that the bracket has been moved, glued back or bent. Then the connections: the connector and wiring, and on many cameras a heater element for the glass. Finally the data: codes in the camera module, the calibration status, and the camera's own diagnostic values, such as image blockage counters and the angles found by self-calibration.

Cleaning the inside of the glass in front of the forward camera on many vehicles requires removing the camera cover, and on some removing the camera itself. Kiran follows the procedure, because unclipping a camera from its bracket may itself be an event that requires calibration on some makes.

# Camera self-check data from a scan, compared with limits (values illustrative)
readings = {"blockage count (last 100 trips)": 14, "pitch found by self-cal (deg)": 0.4,
            "yaw found by self-cal (deg)": 2.6, "glass heater current (A)": 0.0,
            "image temperature (C)": 31}
limits = {"blockage count (last 100 trips)": (0, 20), "pitch found by self-cal (deg)": (-2.0, 2.0),
          "yaw found by self-cal (deg)": (-2.0, 2.0), "glass heater current (A)": (0.4, 2.0),
          "image temperature (C)": (-40, 85)}
for k, v in readings.items():
    lo, hi = limits[k]
    flag = "ok" if lo <= v <= hi else "CHECK"
    print(f"{k:34} {v:6} {flag}")

blockage count (last 100 trips)        14 ok
pitch found by self-cal (deg)         0.4 ok
yaw found by self-cal (deg)           2.6 CHECK
glass heater current (A)              0.0 CHECK
image temperature (C)                  31 ok

The sample readings above point to two separate problems: a yaw angle outside the camera's self-calibration window, suggesting the camera or bracket is aimed off to one side, and a glass heater drawing no current, which would explain morning fog. Each points to a different next test. Neither is solved by clearing codes.