Multimodal models and the Gemini family

Jules arrives at the third session with his phone. Over the weekend he photographed a customer's returned hiking boot, its sole peeling at the toe, and asked an AI assistant what had gone wrong. The assistant described the separation, guessed at the cause and suggested how the return should be coded. "It read a photo and answered in words," he says. "That is not the same as a chatbot." Ravi agrees, and uses the example to introduce multimodal models and the family of models Google offers on Google Cloud.

What multimodal means

A modality is a kind of data: text, images, audio, video and code are each modalities. A multimodal model can take in, and in some cases produce, more than one modality. It can read a photo and a question together and answer in text; listen to a recorded call and summarize it; watch a video and list what happens in each scene; or read a chart in a PDF and explain the trend.

Early AI systems handled modalities separately: one model for speech recognition, another for image labels, another for text, connected by code that passed outputs from one to the next. A natively multimodal model is trained on several modalities together from the start, so it can reason across them directly. When the boot photo and the question "Is this a manufacturing defect or wear?" go in together, the model can relate the visible damage to the words.

For a business, multimodality matters because so much information is not text. Birchline's shop has 140,000 product photos, store managers send pictures of damaged deliveries, the help desk records calls, and suppliers send specifications as scanned PDFs. A multimodal model can work with all of these without first converting each into text by a separate system.

Google's foundation models

Google builds several families of foundation models and offers them on Google Cloud. The Generative AI Leader exam guide names four that leaders should be able to match to use cases: Gemini, Gemma, Imagen and Veo. Model versions and sizes change frequently, so Ravi advises learning what each family is for rather than memorizing version numbers.

Gemini

Gemini is Google's flagship family of multimodal models. Gemini models were designed from the start to work with text, images, audio, video and code, and they can reason across these inputs together. They are offered in several sizes, from larger models suited to complex reasoning, coding and long documents to smaller, faster and cheaper models suited to high-volume tasks such as classification or quick replies. Gemini models support very large context windows, so they can consider long documents, large codebases or long recordings in a single request; lesson 4 explains what that means.

Gemini appears in many places. It powers the Gemini app that consumers use, Gemini in Google Workspace that helps staff write and summarize in Gmail and Docs, Gemini Enterprise for organizations, and coding tools such as Gemini Code Assist. Developers reach Gemini models on Google Cloud through the Gemini Enterprise Agent Platform (earlier materials call it Vertex AI) and through Google AI Studio. GCP-111 tours these products; this course concentrates on what the models can do.

For Birchline, Gemini is the natural candidate for the shopping assistant, the description writer and the review summaries, because these tasks involve language, product photos and long collections of text. Ravi's plan is to test a larger and a smaller Gemini model on each task and pick by measured quality, speed and cost.

Gemma

Gemma is Google's family of lightweight open models, built from the same research and technology as Gemini. Gemma models publish their parameters, so organizations can download them, run them on their own hardware, on Google Cloud or even on laptops and phones, and tune them for specialized tasks under the Gemma terms of use. Because they are smaller, they need less computing power to run.

Gemma suits situations where an organization wants control: running a model where data must not leave a particular environment, tuning a small model deeply for one narrow job, building something that must run offline, or keeping costs predictable at very high volumes. Farah mentions that Birchline's stores have unreliable internet in a few mountain towns, and a small model that runs on a store device is one way to keep a simple product lookup working there.

Imagen

Imagen is Google's family of text-to-image diffusion models. Given a written description, Imagen generates new images; it can also edit images, for example changing a background or extending a scene. Marketing teams use image generation for campaign concepts, mood boards, product backgrounds and variations of a design.

Jules is enthusiastic until Teresa joins the conversation. Generated images raise questions a retailer must answer before using them: whether a generated image could mislead a customer about what a product actually looks like, how generated images are labelled, and how intellectual property is respected. Google applies SynthID, an invisible digital watermark, to images generated by its models so that they can be identified as AI-generated later. Birchline decides that generated images may be used for campaign backgrounds and concepts, never as a substitute for real photographs of the product a customer will receive.

Veo

Veo is Google's family of video generation models. It creates short video clips from text descriptions or from images, with control over style and camera movement, and newer versions can generate sound with the video. Uses include advertising concepts, social media content, storyboards and training material. As with images, generated video carries a SynthID watermark, and the same honesty rules apply: a generated clip of a tent standing up to a storm must not be presented as a real product test.

Google offers other specialized models as well, for example for speech and for music, and Model Garden lists models from other companies too. The four above are the ones leaders are expected to know by name and use case.

Capabilities that cross modalities

Ravi shows Jules three capabilities that only make sense with multimodal models, because they change what Birchline can automate.

Visual question answering. A store manager photographs a shelf and asks which items are missing labels. The model reads the image and answers in text. The same capability can check that a product photo matches its listing, for example that a jacket labelled red is actually red.

Document understanding. Supplier specification sheets arrive as scanned PDFs with tables, logos and handwritten notes. A multimodal model can read the whole page, including the table layout, and extract fields such as fabric, weight and care instructions into structured data. Previously this needed a separate optical character recognition system plus hand-built rules for every supplier's format.

Long-media summarization. A one-hour recorded supplier meeting or a set of help desk calls can be summarized in one request when the model's context window is large enough, with timestamps for the moments that matter.

# Estimate how much of a model's context a mixed request uses.
# The rates below are illustrative planning figures, not published limits.
TOKENS_PER_WORD = 1.3          # rough average for English text
TOKENS_PER_IMAGE = 260         # illustrative cost of one small image
TOKENS_PER_AUDIO_MINUTE = 2000 # illustrative cost of one minute of audio

request = {"instruction words": 120, "spec sheet words": 900,
           "product photos": 4, "call recording minutes": 6}

total = (request["instruction words"] + request["spec sheet words"]) * TOKENS_PER_WORD
total += request["product photos"] * TOKENS_PER_IMAGE
total += request["call recording minutes"] * TOKENS_PER_AUDIO_MINUTE
print("estimated input tokens:", round(total))
for window in (32_000, 128_000, 1_000_000):
    print(f"share of a {window:,}-token window: {total / window:.3f}")

estimated input tokens: 14366
share of a 32,000-token window: 0.449
share of a 128,000-token window: 0.112
share of a 1,000,000-token window: 0.014

The rates are illustrative, and real token costs for images, audio and video depend on the model and its settings, which the provider documents. The point is the habit: before designing a multimodal workflow, estimate how much input each request carries, because input size drives both cost and whether the request fits at all.

Limits of multimodal models

Multimodality widens what models can do, but it does not remove their weaknesses. A model can misread a blurry photo, miscount small objects, misjudge colours under poor lighting, or invent details that are not in an image. Audio transcription degrades with background noise and strong accents. Generated images can contain distorted text or impossible details. And because more kinds of input are sensitive, privacy questions multiply: a photo of a store may include customers' faces, and a recorded call contains a customer's voice and personal details.

Teresa adds two rules to Birchline's draft AI policy as a result. Images and recordings that include customers must be handled under the same privacy rules as other personal information, including PIPEDA and, for Quebec shoppers, Law 25. And any automated decision based on an image, such as approving a return, must be reviewable by a person.

Choosing a size within a family

Within the Gemini family, the choice between a larger and a smaller model is usually a trade between quality on hard tasks and speed and cost on easy ones. Ravi tests both sizes on the same twenty examples of each task. For the review digest, the smaller model misses subtle complaints buried in long reviews, so the larger one earns its price. For tagging product photos with colour and category, the smaller model is as accurate and answers several times faster. Many teams end up using two or three sizes in one application, sending each request to the cheapest model that handles it well, a pattern sometimes called model routing.