Detection and segmentation

Classifying a whole image as "contains a dog" is rarely enough. A self-driving car must know where every pedestrian, vehicle and lane line is; a radiologist's tool must outline a tumour; a warehouse robot must find the exact shape of each parcel to grasp it. Object detection finds and labels each object with a bounding box; semantic segmentation labels every pixel with a class; instance segmentation separates individual objects pixel by pixel; and related tasks find keypoints such as human body joints. This lesson covers how these tasks are framed, the main model families, how they are evaluated, and the special issues raised by detecting and recognizing people.

The tasks

Bounding boxes and IoU

A detector outputs boxes, usually as corner coordinates \((x_1, y_1, x_2, y_2)\) or as centre, width and height. The standard measure of how well a predicted box matches a ground-truth box is intersection over union (IoU):

IoU is 1 for a perfect match and 0 for no overlap. A detection usually counts as correct if its IoU with an unmatched ground-truth box of the same class is at least 0.5, though stricter thresholds (up to 0.95) are used to reward precise localization.

Detector families

Detectors predict boxes at many candidate positions and keep the confident ones. Three generations of design are common:

• Two-stage detectors (R-CNN, Fast R-CNN, Faster R-CNN, 2014 to 2015): a region proposal network suggests a few thousand candidate boxes from the CNN feature map; a second stage classifies each and refines its coordinates. Accurate, and the basis of Mask R-CNN for instance segmentation.
• One-stage detectors (SSD, the YOLO family, RetinaNet): predict classes and box offsets directly at every location of one or more feature maps, often relative to preset anchor boxes of several shapes. Faster, and widely used for real-time detection on video and embedded devices. RetinaNet introduced focal loss, which down-weights the overwhelming number of easy background locations.
• Transformer-based detectors (DETR, 2020, and successors): treat detection as predicting a set of objects, using learned object queries and a matching loss, with no anchors or duplicate removal needed. Modern variants are both accurate and fast.

Most detectors use feature pyramids that combine coarse, semantically strong deep features with fine, high-resolution early features, so that both large and small objects are detected. Small objects remain the hardest case, since they occupy few pixels.

Non-maximum suppression

Dense detectors predict many overlapping boxes for the same object. Non-maximum suppression (NMS) removes duplicates: sort boxes by confidence, keep the highest, discard any remaining box whose IoU with it exceeds a threshold, and repeat.

import numpy as np

def iou(a, b):
    # boxes as (x1, y1, x2, y2)
    ix = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    iy = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = ix * iy
    union = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter
    return inter / union

def nms(boxes, scores, thr=0.5):
    order = list(np.argsort(scores)[::-1])
    keep = []
    while order:
        i = order.pop(0)
        keep.append(int(i))
        order = [j for j in order if iou(boxes[i], boxes[j]) < thr]
    return keep

boxes = [(10, 10, 110, 210), (14, 18, 112, 214), (20, 5, 118, 200), (150, 40, 250, 220), (160, 50, 245, 230), (300, 300, 340, 330)]
scores = [0.92, 0.85, 0.60, 0.88, 0.75, 0.30]
print("IoU(box 0, box 1) =", round(iou(boxes[0], boxes[1]), 3))
print("IoU(box 0, box 3) =", round(iou(boxes[0], boxes[3]), 3))
for thr in (0.5, 0.8):
    print(f"NMS at IoU threshold {thr}: keep boxes {nms(boxes, scores, thr)}")

IoU(box 0, box 1) = 0.887
IoU(box 0, box 3) = 0.0
NMS at IoU threshold 0.5: keep boxes [0, 3, 5]
NMS at IoU threshold 0.8: keep boxes [0, 3, 4, 2, 5]

With the common threshold of 0.5, NMS keeps one box per object (boxes 0, 3 and 5), discarding the near-duplicates around the first two objects. A looser threshold of 0.8 keeps duplicates whose overlap is below 0.8, such as box 4 (an offset second box on the car) and box 2. Too low a threshold, by contrast, merges genuinely separate objects that overlap, such as people in a crowd, which is one reason set-based transformer detectors that avoid NMS are attractive.

Evaluating detectors: precision, recall and AP

Each detection is a true positive if it matches an unmatched ground-truth object at the IoU threshold, or a false positive otherwise; unmatched ground-truth objects are false negatives. Ranking detections by confidence and moving the threshold down traces a precision-recall curve. Average precision (AP) is the area under it (with precision made non-increasing), summarizing performance across all thresholds:

Mean average precision (mAP) averages AP over classes. The COCO benchmark reports AP averaged over IoU thresholds from 0.5 to 0.95, written AP or AP@[.5:.95], as well as AP50 and AP75, and separately for small, medium and large objects.

import numpy as np

# detections for one class across a test set, sorted later by score; True = matched a ground truth at IoU >= 0.5
scores = np.array([0.95, 0.91, 0.88, 0.80, 0.77, 0.70, 0.62, 0.55, 0.43, 0.30])
correct = np.array([1, 1, 0, 1, 1, 0, 1, 0, 0, 1])
n_gt = 8                                   # objects of this class in the ground truth

order = np.argsort(-scores)
tp = np.cumsum(correct[order]); fp = np.cumsum(1 - correct[order])
precision = tp / (tp + fp); recall = tp / n_gt
for k in range(len(scores)):
    print(f"top {k + 1:2d} (score >= {scores[order][k]:.2f}): precision {precision[k]:.2f}, recall {recall[k]:.3f}")

# average precision: area under the precision-recall curve, with precision made non-increasing
p_interp = np.maximum.accumulate(precision[::-1])[::-1]
r = np.concatenate(([0.0], recall))
ap = np.sum((r[1:] - r[:-1]) * p_interp)
print(f"average precision (AP at IoU 0.5) = {ap:.3f}")

top  1 (score >= 0.95): precision 1.00, recall 0.125
top  2 (score >= 0.91): precision 1.00, recall 0.250
top  3 (score >= 0.88): precision 0.67, recall 0.250
top  4 (score >= 0.80): precision 0.75, recall 0.375
top  5 (score >= 0.77): precision 0.80, recall 0.500
top  6 (score >= 0.70): precision 0.67, recall 0.500
top  7 (score >= 0.62): precision 0.71, recall 0.625
top  8 (score >= 0.55): precision 0.62, recall 0.625
top  9 (score >= 0.43): precision 0.56, recall 0.625
top 10 (score >= 0.30): precision 0.60, recall 0.750
average precision (AP at IoU 0.5) = 0.614

As the threshold falls, recall rises to 0.75 (6 of 8 objects found) while precision drifts down to 0.6. The area under the interpolated curve gives an AP of 0.61. Choosing an operating threshold is a separate, application-specific decision: a car's pedestrian detector needs very high recall, while a photo-tagging feature might prefer high precision.

Semantic and instance segmentation

Segmentation networks must output a full-resolution map, but CNN backbones reduce resolution. Fully convolutional networks (FCN, 2015) replaced final fully connected layers with convolutions and upsampled the coarse output. U-Net (2015), developed for biomedical images, uses an encoder that downsamples and a decoder that upsamples, with skip connections passing fine detail from each encoder level to the matching decoder level; it remains a standard for medical and scientific segmentation, and the same shape underlies image diffusion models. DeepLab used dilated (atrous) convolutions to enlarge receptive fields without losing resolution. Instance segmentation adds a mask prediction to each detection, as in Mask R-CNN, or uses query-based transformer models such as Mask2Former that handle semantic, instance and panoptic segmentation in one architecture.

Since 2023, promptable foundation models such as Meta's Segment Anything (SAM) can segment almost any object given a click, box or text prompt, without task-specific training, and are now used to accelerate annotation and as components of larger systems.

Segmentation is evaluated per class with IoU (also called the Jaccard index) or the Dice coefficient \(2|A \cap B|/(|A| + |B|)\), averaged over classes as mean IoU. Pixel accuracy is easy to compute but dominated by large classes such as background.

import numpy as np

# 3 classes on a 10 x 10 image: 0 = background, 1 = road, 2 = pedestrian (small)
gt = np.zeros((10, 10), int); gt[6:, :] = 1; gt[2:6, 4:6] = 2
pred = gt.copy()
pred[6, :] = 0                      # misses the top row of road
pred[2:6, 4] = 0                    # misses half the pedestrian

def metrics(gt, pred, k):
    acc = (gt == pred).mean()
    ious, dices = [], []
    for c in range(k):
        inter = ((gt == c) & (pred == c)).sum()
        union = ((gt == c) | (pred == c)).sum()
        ious.append(inter / union)
        dices.append(2 * inter / ((gt == c).sum() + (pred == c).sum()))
    return acc, ious, dices

acc, ious, dices = metrics(gt, pred, 3)
print(f"pixel accuracy: {acc:.3f}")
for c, name in enumerate(("background", "road", "pedestrian")):
    print(f"{name:10s}: IoU {ious[c]:.3f}  Dice {dices[c]:.3f}  pixels in ground truth {(gt == c).sum()}")
print(f"mean IoU: {np.mean(ious):.3f}")

pixel accuracy: 0.860
background: IoU 0.788  Dice 0.881  pixels in ground truth 52
road      : IoU 0.750  Dice 0.857  pixels in ground truth 40
pedestrian: IoU 0.500  Dice 0.667  pixels in ground truth 8
mean IoU: 0.679

Pixel accuracy (86%) looks good, but the small pedestrian class, which matters most for safety, has IoU of only 0.5 because half its eight pixels were missed. Mean IoU (0.68) weights each class equally and exposes the problem. Safety-critical applications report per-class results and often weight errors on rare, important classes more heavily.

Annotation

Dense prediction tasks need costly labels: drawing a box takes about 7 to 30 seconds, and a precise polygon mask a minute or more per object, so a large segmentation dataset can cost millions of dollars to annotate. Annotation quality, consistency and clear guidelines (what counts as an object, how to handle occlusion) strongly affect model quality. Approaches to reduce cost include model-assisted labelling (a model proposes, a person corrects), promptable segmentation tools, active learning (labelling the examples the model is least sure about), weak supervision from image-level labels, and synthetic data rendered from 3D simulations with perfect labels.

Detecting and recognizing people

Person detection, pose estimation and face analysis are among the most valuable and most sensitive vision applications. Face recognition compares an embedding of a face (Lesson 7) with stored embeddings to verify identity (one-to-one, as when unlocking a phone) or identify a person (one-to-many, searching a database). Accuracy has improved dramatically, but studies such as the US National Institute of Standards and Technology's evaluations have found error rates that vary with demographics for many algorithms, especially in less controlled images, and errors in one-to-many identification have led to wrongful arrests. Surveillance, tracking and biometric identification raise serious privacy and civil liberties concerns.

In Canada, biometric information such as facial templates is sensitive personal information under privacy law. In 2021 the federal Privacy Commissioner and provincial counterparts found that Clearview AI's scraping of billions of online images to build a facial recognition database violated Canadian privacy laws, and that the RCMP's use of it contravened the Privacy Act. Québec's Law 25 requires organizations to disclose the creation of biometric databases to the province's Commission d'accès à l'information in advance and generally to obtain express consent to verify identity with biometrics. Privacy commissioners have issued guidance on police use of facial recognition, and retailers using it have been found in breach of privacy law.

Responsible deployment involves evaluating accuracy across demographic groups and conditions, setting thresholds appropriate to the stakes, keeping humans responsible for consequential decisions, minimizing data collection and retention, and being transparent with the people affected.

Lesson 6 looks inside modern detectors: how positives are assigned, focal loss, DETR, panoptic segmentation and pose estimation.