Computer vision and image generation workloads

For the session on December 16, Grace Oduya, the store operations director, joins Arjun and Maya with a phone full of photographs. Each one shows a shelf in one of the company's 23 stores: paint cans, garden hoses, boxes of screws, a bay of cordless drills with three empty hooks. "Every week, staff walk every aisle to find gaps and misplaced stock," she says. "It takes hours. Could the AI look at the photos instead?" Marcus Feldt has sent a request too: could AI generate lifestyle images of products in a garden or kitchen for the website, instead of paying for a photo shoot each season? This lesson covers both sides of vision: understanding images and creating them.

How computers see

To a computer, an image is a grid of numbers. Each pixel has values for its colour, usually three numbers for the red, green and blue channels, each from 0 to 255. A modest phone photo of 4,000 by 3,000 pixels holds 36 million such numbers. Early computer vision relied on hand-written rules to find edges and shapes in those numbers. Modern vision uses deep learning: neural networks trained on millions of labelled images learn, layer by layer, to detect edges, then textures, then parts of objects, then whole objects. Newer vision models use transformer architectures, the same family that powers language models, and multimodal models combine vision and language in one network.

# A tiny 4 x 4 "image" of brightness values (0 = black, 255 = white): a dark box on a light shelf.
img = [[230, 228, 231, 229],
       [227,  40,  38, 230],
       [229,  42,  41, 228],
       [231, 229, 230, 232]]
pixels = [v for row in img for v in row]
dark = [(r, c) for r, row in enumerate(img) for c, v in enumerate(row) if v < 100]
print("pixels:", len(pixels), " average brightness:", round(sum(pixels) / len(pixels)))
print("dark pixels at (row, column):", dark)
print("numbers in a 4000 x 3000 RGB photo:", 4000 * 3000 * 3)

pixels: 16  average brightness: 182
dark pixels at (row, column): [(1, 1), (1, 2), (2, 1), (2, 2)]
numbers in a 4000 x 3000 RGB photo: 36000000

The main vision tasks

Computer vision is not one task but several, and the difference matters when choosing a service and judging results.

Object detection is the right fit for Grace's shelf photos, because she needs to know not only that drills are present but where the gaps are. A detector returns each object it finds as a label, a confidence score and a bounding box, the rectangle that encloses the object, given as coordinates. Detection quality is judged partly by how well the predicted box overlaps the true box, measured by intersection over union: the area where the two boxes overlap divided by the area they cover together. A score of 1 means a perfect match; 0.5 or more is a common threshold for counting a detection as correct.

# Intersection over union for a predicted box and a true box: (left, top, right, bottom) in pixels.
def iou(a, b):
    w = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    h = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = w * h
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)
true_box = (100, 50, 200, 250)
predicted = (110, 60, 210, 260)
print(f"IoU = {iou(true_box, predicted):.3f}")

IoU = 0.747

Prebuilt and custom vision models

A prebuilt vision model has been trained by Microsoft or another provider on a broad range of everyday images, so it can recognize common objects such as chairs, bottles and people without any work from the customer. But it has never been told what a Harrow & Finch shelf label looks like, or the difference between two nearly identical boxes of screws. When a prebuilt model is not good enough, a team can train a custom model by giving it labelled examples of its own: a few hundred photos of shelves with every product and every empty hook marked by a box. The model starts from what the prebuilt model already knows and learns the new categories from the examples, a technique called transfer learning, so far fewer images are needed than training from scratch would require.

The labelling itself is real work. Someone has to draw boxes around each item in each training photo, and the photos must cover the variety the model will meet in production: bright and dim aisles, phones held at different angles, shelves half hidden by a cart. Grace volunteers two store supervisors to label photos for a week in January. Arjun warns that if all the training photos come from the newest store, with its bright lighting, the model may struggle in the oldest store, where half the aisle lights are older fluorescent tubes. Measuring results store by store will show whether that is happening.

Optical character recognition

Optical character recognition, or OCR, reads printed and handwritten text from images and scanned documents. It is one of the oldest and most reliable vision tasks, and it underpins much information extraction: before a system can find the total on an invoice, it has to read the text. Modern OCR handles photos taken at angles, curved labels and many languages. At Harrow & Finch, OCR could read price labels in shelf photos and compare them with the prices in StockLine, catching labels that were not updated after a price change.

OCR results come with positions and confidence scores, just like detections. A label photographed at an angle, partly covered by a hanging product or printed in a tiny font may be read with low confidence, and the system should treat such readings as uncertain rather than act on them. For the price-label check, Grace agrees that only mismatches read with high confidence will create a task for store staff; the rest will simply be skipped until the next weekly photo.

Faces and sensitive uses

Face technologies need particular care. Detecting that a face is present, so that it can be blurred, is relatively low risk. Identifying who a person is, or inferring attributes such as emotion, raises serious concerns about privacy, consent and accuracy across different groups of people. Microsoft restricts access to its face identification and verification capabilities, requiring customers to apply and meet its responsible AI requirements, and has retired features that inferred emotional states and other sensitive attributes. For the shelf project, Carmen Ortiz rules that photos should be framed to avoid shoppers, and that any faces that do appear will be blurred automatically before images are stored.

Vision in multimodal models

The newest way to use vision is to include an image in a prompt to a multimodal model and ask about it in plain language: "Which hooks in this photo are empty, and what do the labels above them say?" The AI-901 outline expects candidates to interpret visual input in prompts using a deployed multimodal model. This approach is flexible: the same model can count gaps, read labels, describe damage on a returned product from a customer's photo or explain a chart, without training. The trade-offs are the familiar ones from lesson 2. A multimodal model is less predictable than a specialized detector, its counts and positions can be wrong, and it costs more per image. Images are also converted to tokens, often hundreds per image, so they consume context and quota.

Arjun proposes testing both approaches on fifty of Grace's photos: a multimodal model prompted to list empty hooks, and a specialized detector. Whichever finds more of the real gaps, with fewer false alarms, at an acceptable cost per photo, wins.

Image generation

Image generation models create new images from a text description, called a prompt, and some can edit an existing image according to instructions. Most current models work by a process called diffusion: they start from random noise and remove it step by step, guided by the prompt, until an image emerges. Foundry's model catalog includes image generation models, and the AI-901 outline asks candidates to create new visual outputs using generative models.

Marcus's request raises the questions that come with any generated image. Is it accurate? A generated picture of a patio set must not show cushions the product does not include or a colour it is not sold in, or the company risks misleading customers. Who owns it, and could it resemble someone else's work? Should customers be told it is generated? Microsoft attaches content credentials, based on an industry standard from the Coalition for Content Provenance and Authenticity, to images from some of its generation services so that their origin can be checked. The review group decides that generated images may be used for mood and background only, never to depict the product itself, which will still be photographed, and that generated images will be labelled.

Prompts for images follow the same principles as prompts for text: be specific about the subject, setting, style, lighting, composition and what to leave out. "A sunny backyard deck in late afternoon, wide shot, natural light, no people, no text, plain wooden furniture" gives the model far more to work with than "a nice garden". Image models also have content filters that block requests for violent, sexual or otherwise harmful images, and requests that try to imitate real people.

Video

Video is a sequence of images with sound, so video analysis combines vision and speech: detecting objects and scenes over time, reading on-screen text, transcribing speech and summarizing what happens. Azure Content Understanding, covered in the next lesson, can extract information from video. Harrow & Finch's training team, which records product demonstration videos for store staff, would like automatic chapter markers and summaries, a low-risk use that is now on the list for 2027.