How AI sees, hears and reads

Some of the most visible achievements of AI involve perception: unlocking a phone with a face, transcribing a meeting, translating a sign through a camera, reading an X-ray, or guiding a car through traffic. For people, seeing, hearing and understanding language feel effortless, which hides how hard they are for machines. For decades, computer vision, speech recognition and natural language processing were separate fields with hand-crafted methods; deep learning has transformed all three and increasingly merges them into single multimodal models. This lesson explains how computers represent images, sound and text as numbers, how neural networks extract meaning from them, what these systems can now do, and where they still fail.

Images as numbers

To a computer, a digital image is a grid of numbers. A grayscale image stores one brightness value per pixel, typically from 0 (black) to 255 (white); a color image stores three values per pixel, for red, green and blue. A 12-megapixel phone photo is therefore about 36 million numbers. The challenge of computer vision is to turn this huge grid into meaning: there is a cat, the person is smiling, this X-ray shows a fracture, the car ahead is braking.

Filters and convolution

A key operation in computer vision is convolution: sliding a small grid of weights, called a filter or kernel, over the image and computing a weighted sum at each position. Different filters respond to different patterns: a filter with negative weights on the left and positive weights on the right responds strongly where brightness increases from left to right, that is, at a vertical edge. Classic computer vision used hand-designed filters like these to find edges, corners and textures.

import numpy as np

img = np.zeros((8, 8))
img[2:6, 2:6] = 1.0
kernel_v = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]])


def convolve(image, k):
    h, w = image.shape
    out = np.zeros((h - 2, w - 2))
    for r in range(h - 2):
        for c in range(w - 2):
            out[r, c] = np.sum(image[r:r + 3, c:c + 3] * k)
    return out


edges_v = convolve(img, kernel_v)
edges_h = convolve(img, kernel_v.T)
print("image (1 = bright square):")
for row in img.astype(int):
    print("  " + " ".join(str(v) for v in row))
print("vertical-edge filter response:")
for row in edges_v.astype(int):
    print("  " + " ".join(f"{v:3d}" for v in row))
mag = np.sqrt(edges_v ** 2 + edges_h ** 2)
print("pixels marked as edges (magnitude > 2):", int(np.sum(mag > 2)), "of", mag.size)

image (1 = bright square):
  0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0
  0 0 1 1 1 1 0 0
  0 0 1 1 1 1 0 0
  0 0 1 1 1 1 0 0
  0 0 1 1 1 1 0 0
  0 0 0 0 0 0 0 0
  0 0 0 0 0 0 0 0
vertical-edge filter response:
    1   1   0   0  -1  -1
    3   3   0   0  -3  -3
    4   4   0   0  -4  -4
    4   4   0   0  -4  -4
    3   3   0   0  -3  -3
    1   1   0   0  -1  -1
pixels marked as edges (magnitude > 2): 28 of 36

The filter responds with large positive values at the square's left edge (dark to bright) and large negative values at its right edge (bright to dark), and zero in uniform regions. A convolutional neural network (Lesson 6) learns hundreds of such filters from data rather than having them designed by hand: the first layers learn edge and color detectors much like this one, and deeper layers combine them into detectors for textures, parts and whole objects.

What computer vision can do

Vision tasks range from simple to complex:

• Image classification: what is in this image? Networks now exceed human accuracy on some benchmark datasets.
• Object detection: where are the objects, with boxes around each (used in self-driving cars, retail checkout and security).
• Segmentation: which pixels belong to which object (used in medical imaging and photo editing).
• Face recognition: identifying or verifying people, accurate but controversial (Lesson 11).
• Pose estimation and tracking: locating people's joints and following objects over time.
• Optical character recognition (OCR): reading text in images and documents.
• 3D reconstruction and depth estimation, used in robotics, mapping and augmented reality.
• Image generation and editing: creating or modifying images from text descriptions (Lesson 9).

Sound and speech

Sound is a pressure wave, recorded as a sequence of amplitude samples, typically 16,000 per second for speech. Speech systems usually convert audio into a spectrogram, an image-like picture showing how much energy is present at each frequency over time, in which vowels and consonants have recognizable patterns. Neural networks then map spectrograms to text (speech recognition) or text to audio (speech synthesis).

Speech recognition improved dramatically with deep learning. Systems trained on hundreds of thousands of hours of audio, such as OpenAI's Whisper (2022), transcribe many languages with accuracy approaching that of human transcribers in good conditions, though accuracy drops with noise, accents and specialized vocabulary. Speech synthesis has become so natural that cloned voices can be hard to distinguish from real ones, enabling accessibility tools and audiobooks, but also fraud: criminals have used cloned voices to impersonate relatives and executives.

Text as numbers

Computers cannot read words directly. Language systems first split text into tokens (words or word pieces) and map each token to a vector of numbers, called an embedding. Embeddings are learned so that words used in similar contexts get similar vectors: "doctor" ends up near "nurse" and "physician", and "Paris" near "London". Remarkably, directions in the embedding space can capture relationships; in early word embedding models, the vector from "man" to "woman" was similar to the vector from "king" to "queen".

Natural language processing

Natural language processing (NLP) covers tasks such as translation, summarization, question answering, sentiment analysis, information extraction and dialogue. Language is hard for machines because it is ambiguous ("I saw the man with the telescope"), depends on context and world knowledge, and is full of idioms, sarcasm and references. Older systems used grammars and rules, then statistical models trained on large text collections. Neural machine translation, introduced widely around 2016, made online translation dramatically more fluent. Since 2018, transformer-based language models pretrained on vast text collections have taken over almost every NLP task, culminating in the general-purpose assistants described in Lesson 9.

Multimodal models

The boundaries between vision, speech and language are disappearing. Models such as CLIP (2021) learned to match images with their captions by training on hundreds of millions of image-text pairs, so that images and text share one embedding space; this allows searching photos by description and guiding image generators with text. Today's leading AI assistants are multimodal: they can describe photos, read charts and handwriting, answer questions about documents, listen and speak in real time, and generate images. Robots and self-driving systems combine cameras, microphones, lidar and maps with language understanding to follow instructions in the physical world.

Video and sensor fusion

Video adds time to vision. Systems that understand video track objects from frame to frame, estimate motion, recognize actions (a person falling, a forklift reversing) and predict what will happen next. Because video produces so much data, many systems process it at the edge, on cameras or small computers nearby, sending only events or summaries to central servers.

Real-world systems rarely rely on one sense. Sensor fusion combines cameras, radar, lidar, microphones, GPS and inertial sensors so that each covers the others' weaknesses: cameras see color and texture but struggle in darkness and glare; radar measures distance and speed in rain and fog; lidar gives precise 3D shape. Autonomous vehicles, drones, industrial robots and building security systems all fuse multiple sensors, and estimating how confident the combined picture is matters as much as the picture itself. In data centers, computer vision is used for security, for reading analog gauges, and for detecting smoke, leaks or equipment out of place in camera feeds, usually alongside conventional sensors rather than replacing them.

Where perception still fails

Despite superhuman scores on benchmarks, perception systems remain fragile in ways people are not:

• Distribution shift: a model trained on daytime images may fail at night, in snow, or with a different camera; medical models may fail on scans from a different hospital.
• Adversarial examples and spoofing: small stickers on a stop sign or patterns on glasses can fool classifiers.
• Bias: face recognition systems have shown higher error rates for some demographic groups, notably darker-skinned women in a widely cited 2018 study by Joy Buolamwini and Timnit Gebru, leading to wrongful arrests when misused by police.
• Lack of common sense: a system may label objects correctly while misunderstanding the situation entirely.
• Rare events: the long tail of unusual situations, such as a truck carrying traffic lights or a pedestrian in a costume, is where autonomous systems most often struggle.

Careful testing on representative data, human oversight for consequential decisions, and combining multiple sensors and methods are the standard defenses.