Super-resolution and computational zoom

Zoom is where phone cameras are most constrained. A thin body cannot hold a long lens, so phones combine several fixed lenses, crop from high-resolution sensors and use computation to fill the gaps between them. One of the most elegant ideas in computational photography makes the photographer's own shaky hands useful: because each frame of a hand-held burst is shifted by a fraction of a pixel, together the frames sample the scene more finely than any one of them, and an algorithm can reconstruct detail beyond the sensor's native resolution. This lesson explains sampling and aliasing, multi-frame super-resolution and how Google's Super Res Zoom replaced demosaicing with it, crop zoom from large sensors, multi-camera fusion and the honest limits of all of these. Single-image AI upscaling, which invents detail rather than measuring it, is the subject of lesson 8.

Sampling, resolution and aliasing

A sensor samples the continuous image formed by the lens at regularly spaced points. The sampling theorem says that detail can be represented without ambiguity only up to a spatial frequency of half the sampling rate, the Nyquist frequency. For a sensor with pixel pitch \( p \), the Nyquist frequency is:

Detail finer than this is not simply lost; it is aliased, folding back as false, coarser patterns. In a single image, aliasing is an artefact. But across several images shifted by different sub-pixel amounts, the aliased patterns differ, and the differences carry information about the fine detail. Super-resolution exploits exactly this. The Bayer pattern makes the situation worse for colour: red and blue are sampled at half the pixel pitch in each direction, so their Nyquist frequency is half that of a monochrome sensor, which is why demosaicing produces false colour on fine detail.

Multi-frame super-resolution

Imagine four frames, each shifted by exactly half a pixel horizontally, vertically or both. Together they sample the scene on a grid twice as fine as any one frame. If the shifts are known, the samples can be placed on the fine grid and the high-resolution image reconstructed. Real shifts are random rather than ideal, so the algorithm must estimate each frame's sub-pixel shift precisely (the alignment of lesson 3, refined to a fraction of a pixel) and then interpolate the irregularly placed samples onto a regular fine grid, weighting each sample by its distance and by how trustworthy it is.

import numpy as np
rng = np.random.default_rng(420)
N = 400                                         # fine grid
x = np.arange(N)
scene = np.sin(2 * np.pi * x / 6.5) * 0.5 + 0.5    # detail with a period of 6.5 fine samples
step = 4                                         # sensor samples every 4 fine samples
# a single capture: too coarse for this detail (period 6.5 < 2 * 4 = 8), so it aliases
single = scene[0::step]
single_up = np.interp(x, x[0::step], single)
# four captures with sub-pixel shifts of 0, 1, 2, 3 fine samples (as from hand tremor)
shifts = [0, 1, 2, 3]
fused = np.zeros(N); count = np.zeros(N)
for s in shifts:
    samples = scene[s::step] + rng.normal(0, 0.01, len(scene[s::step]))
    fused[s::step] += samples; count[s::step] += 1
fused = fused / np.maximum(count, 1)
err_single = np.abs(single_up - scene)[10:-10].mean()
err_fused = np.abs(fused - scene)[10:-10].mean()
print(f"single capture error: {err_single:.3f}")
print(f"four shifted captures error: {err_fused:.3f}")
print(f"improvement: {err_single / err_fused:.0f} times")

single capture error: 0.266
four shifted captures error: 0.008
improvement: 35 times

A single coarse capture cannot represent detail with a period shorter than two sensor samples, so interpolating it gives large errors (the pattern is aliased into a false, slower wave). Four captures with different sub-pixel shifts fill the fine grid and recover the pattern with small error. In real images, shifts are irregular, alignment is imperfect and lens blur limits the finest detail, so practical gains are smaller, typically a resolution improvement of about 1.5 to 2 times rather than unlimited.

Super Res Zoom: replacing demosaicing

In 2019 Bartlomiej Wronski and colleagues at Google published "Handheld Multi-Frame Super-Resolution", the method behind the Pixel's Super Res Zoom. Its key insight was that a hand-held burst of raw Bayer frames, each shifted randomly, provides red, green and blue samples at many different positions. Instead of demosaicing each frame and then merging, the algorithm places every raw colour sample directly onto a fine output grid and reconstructs full colour from the accumulated samples. This performs demosaicing and super-resolution in one step and largely avoids demosaicing artefacts. The method uses an anisotropic kernel that adapts to local edges, so samples are spread along edges rather than across them, and a robustness model that rejects samples from moving or misaligned regions. When the phone is on a tripod and there is no natural tremor, the designers used the optical image stabilization system to move the lens slightly in a small pattern, creating the shifts deliberately.

Dedicated cameras do something similar with sensor-shift high-resolution modes, covered in PHOT-201: on a tripod, the sensor is moved by exact half- or one-pixel steps, recording each colour at each location or doubling the sampling grid. Several cameras now offer hand-held high-resolution modes that rely on natural tremor like the phone method. Both approaches need a static scene; moving leaves, water and people produce artefacts unless the software detects and handles motion.

Crop zoom from high-resolution sensors

Many phones have main sensors of 48 to 200 megapixels, usually binned to 12 or 50 megapixels for normal shots. Cropping the central part of the full-resolution readout gives a narrower field of view without upscaling, and manufacturers market this as "optical-quality" 2x zoom. Whether it truly matches an optical 2x lens depends on light: the full-resolution readout of a quad-Bayer sensor requires remosaicing and is noisier per pixel than binned output, so in dim light crop zoom degrades faster. Cropping a quarter of the area also uses only a quarter of the light collected by the whole sensor, so noise in the cropped image is higher than in the full frame at the same output size.

mp_full = 50.0
for zoom in (1, 1.5, 2, 3, 5, 10):
    remaining = mp_full / zoom ** 2
    light_share = 1 / zoom ** 2
    print(f"{zoom:4.1f}x crop: {remaining:5.1f} MP remain, {light_share * 100:5.1f} % of the sensor's light")

 1.0x crop:  50.0 MP remain, 100.0 % of the sensor's light
 1.5x crop:  22.2 MP remain,  44.4 % of the sensor's light
 2.0x crop:  12.5 MP remain,  25.0 % of the sensor's light
 3.0x crop:   5.6 MP remain,  11.1 % of the sensor's light
 5.0x crop:   2.0 MP remain,   4.0 % of the sensor's light
10.0x crop:   0.5 MP remain,   1.0 % of the sensor's light

Multi-camera fusion

Phones typically carry an ultra-wide, a main and one or two telephoto cameras, often a periscope design that folds the light path sideways to fit a longer focal length in a thin body. When the user zooms to a value between two lenses, say 3x on a phone with 1x and 5x cameras, the phone crops from the main camera, possibly enhanced by super-resolution, and may fuse in detail from the telephoto camera where their fields overlap, at the centre of the frame. Fusion needs careful alignment because the two cameras are offset and see the scene from slightly different positions (parallax) and have different colour and noise characteristics. In low light the phone may refuse to use the telephoto, which usually has a smaller sensor and slower aperture, and crop from the main camera instead, sometimes surprising users who thought they were using the telephoto lens. Professional users who need to know which lens was used can check metadata, which often records the actual camera module.

Long zoom, AI and the Moon problem

At extreme zoom, 30x to 100x, the light and detail available are tiny. Phones combine telephoto optics, crop, multi-frame super-resolution and increasingly learned enhancement. This is where the line between measurement and invention is most often crossed. The 2023 controversy over a phone's Moon photos, mentioned in lesson 1, arose because a model trained to recognize the Moon added texture consistent with the Moon's known appearance. Similar learned enhancement is applied to text, faces and architecture at high zoom by several manufacturers. The output looks sharp, but some detail is predicted from training data rather than recorded. For a holiday snap this is harmless; for a wildlife record, a legal photograph of a licence plate or a scientific observation, it can be misleading. Some phones now label or allow disabling of such enhancement, and some include content credentials noting AI processing (lesson 11).

Testing resolution claims

Photographers evaluating zoom should test rather than trust marketing. Photograph a resolution chart or a page of fine text at each zoom setting, in good light and dim light, and compare with a dedicated camera. Shoot fine natural textures such as foliage and brick, which reveal smearing from noise reduction and repeating patterns that can fool alignment. Check metadata to see which camera module was used. Compare single-frame raw captures with the processed output to see how much the pipeline adds. A simple metric is the modulation transfer function (MTF), which describes how much contrast is retained at each spatial frequency; MTF50, the frequency at which contrast falls to half, is a common single-number summary. Software such as Imatest or open-source tools such as MTF Mapper can measure it from slanted-edge targets.

The caution in that answer matters. Sharpening and learned enhancement can raise measured MTF without adding real information. Good testing examines detail that the algorithm cannot predict, such as random text or irregular textures, and compares against a reference.