From optics to computation

For most of photography's history, the image was made almost entirely by optics and chemistry. A lens focused light, film recorded it, and the darkroom interpreted it. Today most photographs are taken on phones, and a phone photograph is better described as a calculation than as a single exposure. The camera captures a burst of raw frames, aligns them, merges them, estimates depth, recognizes faces and skies, adjusts each region differently and delivers a finished picture in a fraction of a second. This is computational photography: the use of algorithms, often including machine learning, to produce images that no single exposure through the same optics could deliver. This course explains how those algorithms work, where artificial intelligence (AI) fits in, what generative AI changes, and how photographers can work honestly and professionally with all of it. PHOT-201 introduced sensors, demosaicing and the basic processing pipeline, and PHOT-202 covered editing; this course builds on both and goes inside the algorithms. Numbers in examples are illustrative unless a source is given.

Why phones need computation

A phone camera faces hard physical limits. Its sensor is small, often around 1/1.3 inch for a main camera and far smaller for ultra-wide and telephoto modules, compared with 36 by 24 mm for a full-frame camera. Its lens must fit in a body less than a centimetre thick, so focal lengths are short and apertures cannot be physically large. Smaller sensors collect less light in total, so images are noisier at the same exposure. Short focal lengths give deep depth of field, so backgrounds rarely blur naturally. Tiny lenses limit optical zoom. A small sensor also has less dynamic range per exposure because each pixel holds fewer electrons before it saturates.

Computation attacks each of these limits. Merging many frames increases the effective amount of light collected, which reduces noise and extends dynamic range. Depth estimation and simulated blur imitate the shallow depth of field of a large sensor. Combining frames shifted by hand tremor recovers detail beyond the native resolution of the sensor. Machine learning models recognize scene content and adjust processing accordingly. None of this changes physics: the light that reached the sensor is still the only real information. Computation is about extracting that information more completely and presenting it more pleasingly, and, increasingly, about filling gaps with learned guesses.

The pipeline from photons to pixels

Every digital camera, phone or not, runs an image processing pipeline. The details are proprietary and change often, but the stages are broadly similar.

• Capture: the sensor converts photons into electrons in each photosite, and an analogue-to-digital converter turns the charge into numbers. Phones typically capture a continuous stream of frames into a ring buffer even before you press the shutter, so the frames used may come from just before the press.
• Raw processing: black level subtraction, correction for defective pixels, lens shading correction and linearization produce a clean raw image in which values are proportional to light.
• Multi-frame processing: several frames are aligned and merged to reduce noise, extend dynamic range or increase resolution.
• Demosaicing: the colour filter array leaves each pixel with only one colour sample; demosaicing estimates the missing two.
• Colour processing: white balance and a colour correction matrix map the sensor's colour response to a standard colour space.
• Tone mapping: the high dynamic range data is compressed into the limited range of a display or file, often with local adjustments.
• Semantic and AI processing: neural networks segment the scene into regions such as sky, skin, hair and foliage and treat each differently, and may denoise, sharpen or upscale.
• Encoding: the image is compressed into JPEG, HEIF or another format, with metadata and, increasingly, a signed provenance record.

Two kinds of hardware share this work. The image signal processor (ISP) is a fixed-function circuit designed to do classic operations very quickly with little power. The neural processing unit (NPU) runs machine learning models. The main processor and graphics processor handle the rest. Much of the progress in phone photography since about 2016 has come from software and these accelerators rather than from optics.

A short history of computational photography

Computational photography grew out of research in computer vision and computer graphics. Paul Debevec and Jitendra Malik's 1997 work on recovering high dynamic range radiance maps from several exposures laid groundwork for HDR photography. Researchers at Stanford, MIT and elsewhere developed light-field cameras, coded apertures and flash/no-flash fusion in the 2000s. The idea of a "camera that computes" moved into consumer products in the 2010s. Google's HDR+ burst pipeline, described in a 2016 paper by Samuel Hasinoff and colleagues, merged several deliberately underexposed frames to reduce noise and preserve highlights. Apple introduced Portrait mode with simulated depth of field in 2016, Google's Night Sight arrived in 2018, and multi-frame super-resolution for zoom followed in 2018 and 2019. Since then, machine learning has spread through every stage, and from about 2022 generative AI added the ability to create or replace image content outright.

Dedicated cameras adopted computation more slowly but now use it too: in-camera HDR, pixel-shift high-resolution modes, focus stacking, live composite modes for night photography, subject-recognition autofocus driven by neural networks and AI-based noise reduction in raw converters. The difference is one of emphasis. A dedicated camera's large sensor and lens collect enough light that computation is optional; a phone depends on it.

Raw, processed and computational raw

A raw file records sensor values with minimal processing, as PHOT-201 explained. On a phone, the meaning of "raw" is complicated. A single-frame raw from a phone sensor is noisy and has limited dynamic range. Several manufacturers therefore offer a computational raw: a file in a raw format, typically Adobe's Digital Negative (DNG), that contains the result of multi-frame merging but has not yet been demosaiced, tone mapped or sharpened, or has been demosaiced but kept linear. Apple ProRAW and Google's raw output from merged bursts are examples. These files give editors the low noise of merging with much of the flexibility of raw.

For photographers this means three choices on a phone: the processed image (JPEG or HEIF), with all of the manufacturer's choices baked in; a computational raw, which keeps merging benefits but leaves tone, colour and sharpening to you; and a single-frame raw from a third-party app, which is the most "honest" sensor record but the noisiest. Understanding the pipeline helps you choose. For documentary work where you must explain what processing occurred, knowing which stages ran is part of professional practice, as lesson 11 discusses.

What computation can and cannot do

It is useful to separate three things computation does. First, it can measure better: combining frames averages away random noise, and that improvement is real information gathered over a longer total time. Second, it can interpret: tone mapping and colour choices decide how measured light looks, and different choices produce different but equally truthful renderings. Third, it can invent: learned models fill in detail that was never captured, such as texture during upscaling, a blurred background in Portrait mode, or new objects in generative fill. Invention can be convincing, but it is not measurement.

These categories overlap. AI denoising, for example, mostly measures and interprets, but at high strength it begins to invent plausible texture. A famous controversy illustrates the line: in 2023 a user photographed a deliberately blurred image of the Moon on a monitor with a phone's high zoom and the phone produced a crisp Moon with crater detail that was not in the source. The manufacturer explained that a scene-optimization model recognized the Moon and enhanced detail. Whatever one thinks of it, the result included learned detail rather than captured detail. Professionals need to know when their tools cross from measurement to invention, because clients, editors, courts and audiences may care.

Signal, noise and the photon budget

Most of computational photography is about noise. Light arrives as discrete photons, and the number arriving at a pixel in a given time varies randomly even under perfectly steady light. This photon shot noise follows Poisson statistics: if a pixel receives on average \( N \) photons, the standard deviation is \( \sqrt{N} \). The signal-to-noise ratio (SNR) from shot noise alone is therefore:

More light means better SNR, but only by the square root. Collecting four times as many photons doubles the SNR. Sensors add further noise, notably read noise from the electronics, which matters most in the darkest tones. A small pixel collects fewer photons than a large one in the same light, so its SNR is lower. Merging frames is a way to collect more photons without a longer single exposure, which would blur with hand or subject motion.

Compute budgets and file sizes

Computation is not free. A phone must finish processing quickly enough that the photographer can take the next picture, and it must not drain the battery or overheat. Engineers therefore work within a compute budget for each photo. Data volumes are large: a 50-megapixel sensor read at 12 bits produces about 75 megabytes of raw data per frame, and a burst of ten frames is 750 megabytes to move and process. Many phones capture at a binned resolution, combining four or more pixels into one, and use full resolution only in good light.

megapixels = 50
bits = 12
frames = 10
bytes_per_frame = megapixels * 1e6 * bits / 8
burst = bytes_per_frame * frames
print(f"one raw frame: {bytes_per_frame / 1e6:.0f} MB")
print(f"burst of {frames}: {burst / 1e6:.0f} MB")
# binning 2x2 reduces pixel count by four
binned = burst / 4
print(f"2x2 binned burst: {binned / 1e6:.0f} MB")
# if the pipeline must finish in 0.5 s, required throughput:
for t in (0.5, 1.0, 2.0):
    print(f"finish in {t} s needs {binned / 1e9 / t:.2f} GB/s")

one raw frame: 75 MB
burst of 10: 750 MB
2x2 binned burst: 188 MB
finish in 0.5 s needs 0.38 GB/s
finish in 1.0 s needs 0.19 GB/s
finish in 2.0 s needs 0.09 GB/s

The output shows why binning and dedicated hardware matter. Even after binning, moving and processing hundreds of megabytes in under a second requires memory bandwidth and specialized circuits. This budget shapes every algorithm in the following lessons: the cleverest method is useless if it takes ten seconds per shot.

Who makes the choices

A computational image embodies thousands of decisions made by engineers, image quality teams and product managers long before you press the shutter. They decide how bright faces should be, how blue skies should look, how much noise is acceptable and how sharp edges should appear. These choices differ between manufacturers and change with software updates, which is why the same scene can look quite different on two phones. Some brands favour a bright, punchy look; others aim for more natural contrast. Some offer "styles" that let users adjust the look before capture, applied within the pipeline rather than as a filter afterward.

For professional photographers the lesson is to treat the phone as an opinionated collaborator. When consistency matters, such as a product catalogue shot partly on a phone, lock what you can (exposure, white balance, style), shoot computational raw where possible, and test how the pipeline treats your typical subjects. When authenticity matters, know what the device did and be ready to explain it.

The structure of this course

The course moves from measurement toward invention. Lessons 2 to 7 cover classic computational photography: the image signal processor, burst capture, HDR and tone mapping, depth and synthetic bokeh, night modes and super-resolution. Lesson 8 covers AI enhancement: denoising, sharpening and upscaling with neural networks. Lessons 9 and 10 cover generative AI: how diffusion and related models work, and how professionals use generative editing ethically and within client, contest and stock rules. Lesson 11 covers authenticity, provenance, content credentials, detection and the law in Canada and the United States. Lesson 12 is a capstone in which you design a computational pipeline and write a provenance and AI-use policy for a photography practice. Several lessons include short Python programs using NumPy with seeded random numbers, so the outputs shown are reproducible and you can experiment by changing parameters.