Night modes and low-light stacking

Night photography used to mean a tripod, a long exposure and patience. Since Google's Night Sight in 2018 and similar modes from every major phone maker, a hand-held phone can make a clean, colorful picture by streetlight or even moonlight. Night modes push the burst ideas of lesson 3 to their limit: they meter motion as well as light, choose how many frames and how long each should be, merge tens of frames with robust alignment, and then apply white balance and tone mapping designed to keep a night scene looking like night. This lesson works through the exposure budget, the trade-off between many short frames and fewer long ones, the special case of astrophotography modes, calibration with dark frames, and the techniques dedicated-camera photographers use for stacking and live composites.

The exposure budget at night

In dim light the only way to collect enough photons is time. A single long exposure would blur from hand shake and subject motion. A night mode therefore divides the total time it can afford into many shorter frames, each short enough to keep blur acceptable, and merges them. Two decisions follow: the duration of each frame and the number of frames. Google's Night Sight, described by Orly Liba and colleagues in 2019, introduced motion metering: before capture, the phone measures how much the scene and camera are moving using optical flow and the gyroscope. If the phone is braced or on a tripod and nothing moves, it uses fewer, longer frames; if the subject moves, it uses more, shorter frames. The published design allowed frames up to about a third of a second hand-held and longer when stable, with total capture up to several seconds (figures from the 2019 description; current phones differ).

Short frames, long frames and read noise

Why not use very short frames always, since they freeze motion? Because each frame adds read noise. If the total exposure time is fixed, splitting it into more frames leaves shot noise unchanged (the same total photons) but adds the read noise of every frame. In bright conditions read noise is negligible compared with shot noise, so splitting costs nothing. In very dim light, where each frame collects only a few photons per pixel, read noise can dominate, and fewer, longer frames give a cleaner result. Night modes balance motion blur against read noise using exactly this reasoning.

Here \( S \) is the signal per frame in electrons, \( n \) the number of frames and \( \sigma_r \) the read noise in electrons per frame. For a fixed total signal \( nS \), the read noise term grows with \( n \).

import numpy as np
total_time = 4.0
read_noise = 1.5                                  # electrons per frame (modern phone sensor, illustrative)
for flux in (100.0, 5.0, 0.5):                    # electrons per second per pixel: bright, dim, very dim
    line = []
    for n in (1, 4, 16, 64):
        S = flux * total_time / n
        snr = n * S / np.sqrt(n * S + n * read_noise ** 2)
        line.append(f"{n:2d} frames SNR {snr:5.2f}")
    print(f"flux {flux:5.1f} e/s: " + " | ".join(line))

flux 100.0 e/s:  1 frames SNR 19.94 |  4 frames SNR 19.78 | 16 frames SNR 19.16 | 64 frames SNR 17.15
flux   5.0 e/s:  1 frames SNR  4.24 |  4 frames SNR  3.71 | 16 frames SNR  2.67 | 64 frames SNR  1.56
flux   0.5 e/s:  1 frames SNR  0.97 |  4 frames SNR  0.60 | 16 frames SNR  0.32 | 64 frames SNR  0.17

At high flux the SNR barely changes with frame count, so short frames are free. At very low flux, splitting into 64 frames loses much of the SNR, because each frame's 1.5 electrons of read noise swamps its tiny signal. This is why astrophotography modes use long individual frames on a tripod.

Merging, color and tone at night

Night merging uses the same alignment and robust merging as lesson 3, but noise is far higher, so distinguishing motion from noise is harder. Pipelines use the noise model to set rejection thresholds, more aggressive spatial denoising where few frames could be merged, and learned denoisers after merging. White balance is difficult because night scenes mix sodium, LED, neon and moonlight, and because a fully neutral correction looks unnatural: our eyes do not fully adapt in the dark, so a perfectly corrected night scene looks like daylight. Night Sight's designers described a learning-based AWB trained on night scenes and a tone mapping strategy that keeps shadows dark and adds contrast so that the result reads as night, drawing on techniques painters use to depict darkness, such as deep blue shadows and strong contrast.

The results raise an interpretive question: a night mode can show colors and detail the eye could not see at the scene. That is measurement (more photons collected) combined with interpretation (how dark to render it). Photographers using night modes for documentary work should be aware that the image may look brighter and more colorful than the scene appeared, and may represent several seconds rather than an instant.

Astrophotography modes

In 2019 Google added an astrophotography mode for Pixel phones on a tripod: up to 15 frames of up to 16 s each, about four minutes in total (as described at the time). Each frame is short enough that stars do not visibly trail, given the wide lens. The pipeline merges the frames, detects and removes hot pixels, which become obvious at long exposures, and uses a segmentation network to find the sky and darken it slightly to look natural. Other manufacturers offer similar modes. On dedicated cameras, the same idea is done manually: shoot many exposures of the night sky, then align on the stars and stack them in software such as Siril, DeepSkyStacker or Sequator, which also handles the foreground separately because the stars move relative to the land.

Calibration frames

Long exposures reveal sensor defects. Hot pixels glow; amplifier glow brightens corners on some sensors; dust and vignetting darken parts of the frame. Astrophotographers calibrate with extra frames. Dark frames are taken with the lens covered at the same exposure time and temperature, and their average is subtracted to remove fixed-pattern noise and hot pixels. Flat frames are taken of an evenly lit surface and used to divide out vignetting and dust shadows. Bias frames, the shortest possible exposures, record the readout offset. Many cameras offer long-exposure noise reduction, which takes an automatic dark frame after each long exposure, doubling the time. Phones apply factory calibration and hot-pixel detection automatically.

import numpy as np
rng = np.random.default_rng(420)
h, w = 100, 100
fixed = rng.gamma(1.0, 2.0, (h, w))                     # fixed-pattern dark signal per pixel
hot = rng.random((h, w)) < 0.002
fixed[hot] += 300                                       # a few hot pixels
sky = 40.0
def expose(signal):
    return rng.poisson(signal + fixed) + rng.normal(0, 2, (h, w))
lights = [expose(sky) for _ in range(10)]
darks = [expose(0.0) for _ in range(20)]
master_dark = np.mean(darks, axis=0)
stack_raw = np.mean(lights, axis=0)
stack_cal = np.mean([l - master_dark for l in lights], axis=0)
print(f"hot pixels: {hot.sum()}")
print(f"uncalibrated stack: mean {stack_raw.mean():.1f}, max {stack_raw.max():.0f}, std {stack_raw.std():.2f}")
print(f"dark-subtracted stack: mean {stack_cal.mean():.1f}, max {stack_cal.max():.0f}, std {stack_cal.std():.2f}")

hot pixels: 18
uncalibrated stack: mean 42.5, max 356, std 13.08
dark-subtracted stack: mean 40.0, max 55, std 2.25

Before calibration, hot pixels reach hundreds of units and fixed pattern adds variation; after subtracting a master dark built from 20 frames, the mean returns to the true sky level, the hot pixels disappear and the remaining variation is mostly random noise. Averaging many darks matters: a single dark frame would add its own random noise to every light frame.

Live composite and creative stacking

Stacking also enables creative effects. Star trails are made by stacking many exposures with a "lighten" blend that keeps the brightest value at each pixel, so each star's path accumulates into an arc. Some cameras, notably OM System and earlier Olympus models, offer live composite modes that build such a lighten stack in camera and show it growing on the screen, which also suits light painting and fireworks, because the background does not overexpose as it would in one long exposure. Average stacking smooths moving water and clouds into a long-exposure look without a neutral density filter, as several phones and cameras offer in "long exposure" or "motion" modes. Maximum and average stacks of the same burst produce very different pictures, and both are honest descriptions of the scene over time if disclosed.

Low light and the limits of honesty

Night modes are among the most impressive and least understood computational features. A phone image of a dark street may show colors and textures no one at the scene could see, people slightly smeared by movement and lights rendered with a chosen warmth. None of this is deceptive in ordinary use. But for evidence, journalism and scientific documentation, record what mode was used, keep the original files and, where it matters, capture a conventional exposure as well. Lesson 11 returns to how provenance records can document such processing automatically.