Cameras, depth cameras and radar

Cameras are the richest and cheapest sensors a robot can carry. A single image holds color, texture, text, faces and the shapes of thousands of objects, and modern vision algorithms turn those pixels into detections, maps and grasp points. Depth cameras add a distance to each pixel, and radar sees through rain, fog and dust that defeat optical sensors. This lesson treats these devices as sensors: how they form their measurements, what specifications matter and where they fail. The geometry of image formation, calibration and the algorithms built on top belong to ROB-330, Computer Vision for Robotics; here the goal is to choose and use the hardware well.

The camera as a sensor

A digital camera focuses light through a lens onto an image sensor, a grid of photosites that each convert incoming photons into charge. Almost all robot cameras use CMOS sensors, in which each pixel has its own amplifier and the image is read out row by row. Color is usually obtained with a Bayer filter: a mosaic of red, green and blue filters over the pixels, with twice as many green as red or blue, followed by demosaicing to estimate full color at every pixel. Monochrome cameras skip the filter and collect roughly three times more light, which is why many machine-vision and stereo cameras are monochrome.

The specifications that matter for robotics are resolution (pixels across and down), frame rate, field of view, which depends on lens focal length and sensor size, pixel size (larger pixels collect more light and give less noise in dim scenes), dynamic range (the ratio between the brightest and darkest detail captured in one image, important when a robot drives from a dark warehouse into sunlight), and the shutter type.

In these formulas w is the sensor width, f the focal length, p the pixel pitch and r the distance to the scene. The ground sample distance (GSD) is the size of the patch of the world that one pixel covers, the camera's equivalent of lidar beam spacing.

Rolling and global shutters

How the sensor exposes its pixels matters enormously for a moving robot. A rolling shutter exposes and reads each row in turn, so the top of the image is captured a few milliseconds before the bottom. Moving objects lean, propellers bend into strange arcs, and fast camera rotations skew the whole image. A global shutter exposes every pixel at the same instant, so the image is a true snapshot. Global-shutter sensors cost more and traditionally have smaller dynamic range, but they are preferred for visual odometry, fast manipulation and anything where geometry must be exact. If a rolling-shutter camera is used, the readout time must be known and modeled.

Motion blur is a separate effect: during the exposure time each pixel integrates light from a moving scene. Keeping blur below about one pixel requires an exposure shorter than the time for the image to move one pixel, which in dim light forces a trade-off between blur and noise. Many robots add illumination, often infrared, to allow short exposures.

Depth cameras

Depth cameras produce an image in which each pixel holds a distance. Three principles are in common use, and each has a distinctive error pattern.

Stereo cameras use two cameras a known distance apart, the baseline b. A point in the scene appears in slightly different horizontal positions in the two images; this difference, the disparity d in pixels, is inversely proportional to depth. Passive stereo needs texture to match, so blank walls give holes; active stereo adds a projected infrared dot pattern to create texture. Stereo works outdoors in sunlight, but its depth error grows with the square of distance.

Structured-light cameras project a known pattern, such as stripes or a coded dot field, and compute depth by triangulating how the pattern deforms on surfaces. They give dense, accurate depth at short range indoors and are used for bin picking and inspection, but sunlight washes the pattern out.

Time-of-flight cameras flood the scene with modulated infrared light and measure, at each pixel, the phase shift of the returning signal, which gives distance. They produce depth for every pixel without needing texture and are fast, but suffer from multipath (light bouncing between surfaces in a corner inflates the depth), flying pixels at object edges, interference from sunlight, and phase wrapping: with modulation frequency \( f_m \), distances beyond \( c/(2f_m) \) alias back to short ranges.

Radar for robots

Radar (radio detection and ranging) works like lidar with radio waves. Because the wavelength is millimeters rather than nanometers, radar passes through rain, snow, fog, dust and smoke almost unaffected, works in total darkness and is cheap to mass-produce. Automotive radars at 76–81 GHz measure range and, through the Doppler effect, the radial velocity of each target directly, which is why every adaptive cruise control system uses one. Compact 60 GHz radar-on-chip modules are now used in robots and buildings for presence detection, people counting and even monitoring breathing.

Most robot and vehicle radars are FMCW: they transmit a chirp whose frequency rises linearly, mix the echo with the transmitted signal and measure the beat frequency, which is proportional to range. The range resolution depends only on the chirp bandwidth B, and the velocity follows from the Doppler shift.

Radar's weakness is angular resolution. Angle is measured with an array of antennas, and with only a few virtual channels the beam is several degrees wide, so a radar point cloud is sparse and blurry compared with lidar. Metal objects return strong echoes while people and animals return weak ones, and reflections off guardrails and floors create ghost targets. Newer imaging radars with hundreds of virtual channels are narrowing the gap. In robotics, radar is valued as a complement: when snow or dust blinds the lidar and cameras, the radar still sees.

Radar and other radio sensors are radio transmitters. In the United States, they must be authorized by the Federal Communications Commission (FCC), for example under Part 95 Subpart M for 76–81 GHz radar, Part 15 rules for 57–71 GHz devices and Part 15 Subpart F for ultra-wideband devices, and carry an FCC ID. Buying a certified module and keeping its antenna and settings unchanged keeps the authorization valid.

Choosing among perception sensors

Cameras give the most information per dollar but need computation and good light; lidar gives precise geometry independent of light but costs more and struggles in weather; radar gives velocity and weather robustness but coarse angle. The industry debate about whether self-driving cars need lidar is really an argument about how much computation and data can substitute for direct measurement. For a robot designer, the practical rule is to start from the failure modes that matter in the operating environment, a dusty mine, a glass-walled office, a snowy parking lot, and choose a set of sensors whose weaknesses do not overlap.