Foundation model types and modalities

The marketing coordinator at Cascade Ridge, Leila Haddad, had heard that the AI hour was "about chatbots" and came to argue the point. She did not want a chatbot. She wanted a way to make simple illustrations for the clinic newsletter, short captions for the waiting-room screens, and perhaps one day a friendly recorded voice for appointment reminders. "Is that the same technology?"

"It is the same family of technology," Noor said, "but different models. A model that writes text is not the model that paints pictures." This lesson sorts foundation models by what they take in and what they produce, called their modalities. It covers large language models, embedding models, image generation and the diffusion technique behind it, multimodal models that understand images and text together, and audio and video models. It then looks at other ways the exam expects you to tell models apart: their size, whether their weights are open, and whether they are general or specialized.

Modalities: what goes in and what comes out

A modality is a kind of data: text, images, audio, video, or structured data such as tables. Describing a model by its input and output modalities is the quickest way to know what it can do. A text-to-text model reads text and writes text. A text-to-image model reads a description and produces a picture. A model that accepts both an image and a question and answers in text is multimodal.

Large language models

Large language models are text-to-text foundation models, covered in lesson 3. They summarize, draft, classify, extract information, translate, answer questions and write code. Most of what people call "chatbots" are interfaces to an LLM. Many current LLMs are also trained to call tools, return structured output such as JSON, and follow long instructions, which makes them the engine of most generative AI applications.

LLMs come in sizes. Smaller models answer faster and cost less per token but handle complex reasoning less well. Larger models are more capable, slower and more expensive. Model families usually offer several sizes for this reason: Amazon's Nova family, for example, has understanding models from a fast, text-only small model up to larger multimodal ones, and other providers in Amazon Bedrock offer similar ranges. Lesson 10 covers how to choose between sizes.

Embedding models

Embedding models take text, and sometimes images, and output a vector rather than new content, as lesson 2 explained. They are foundation models too, trained on broad data, but their job is to represent meaning so that other parts of an application can search, cluster, compare or recommend. In a question-answering system over the procedure manual, the embedding model finds the relevant chunks and the language model writes the answer. Amazon offers embedding models in Bedrock, including the long-standing Titan text embeddings and newer multimodal embeddings, and other providers such as Cohere offer theirs.

Image generation and diffusion models

Most image generation models are diffusion models. The idea is easiest to understand in reverse. During training, the model is shown real images that have been gradually covered in random noise, and it learns to predict and remove a small amount of that noise at each step. To generate an image, the model starts from pure noise and removes noise step by step, guided by the text description, until a picture that matches the description emerges.

Diffusion models can also edit existing images: replacing a background, extending an image beyond its borders, or changing one region while leaving the rest. Amazon's image model in Bedrock is Amazon Nova Canvas; other providers, such as Stability AI, offer theirs. Nova Canvas and similar models add an invisible watermark to generated images so they can later be identified as AI-generated, and support content filters that block harmful requests.

For Leila's newsletter illustrations, Noor suggested a few house rules that the group later adopted: generated images are labeled as illustrations, never presented as photographs of real patients or real clinics, and never used to show medical conditions or treatments, where an invented image could mislead a client.

Multimodal models

A multimodal model accepts more than one modality as input, most commonly images and text, and sometimes video, audio or documents. You can give it a photograph of a pet food label and ask which ingredients are listed, or a scanned referral letter and ask for the referring vet's name and the reason for referral. Several Nova understanding models and models from other providers in Bedrock are multimodal: they take text, images, documents and in some cases video as input and produce text as output.

Multimodal models open uses that earlier needed separate systems. The radiograph image-quality idea from the machine learning sessions, checking whether an X-ray is blurry or mis-positioned before the patient leaves, could one day be explored with a multimodal model. Noor repeated the group's rule: such a model may flag an image for a retake, but reading the radiograph remains a veterinarian's job.

Audio, speech and video models

Generative models also produce sound and moving images. Speech models turn text into natural-sounding voice or carry on spoken conversations; Amazon's Nova Sonic is a speech-to-speech model for real-time voice conversations in several languages. Video models produce short clips from text or images; Amazon's Nova Reel is one. Music and sound-effect generators exist too, though they are less common in business settings.

These sit alongside older AWS AI services that are not foundation models but do related jobs: Amazon Polly converts text to speech with prebuilt voices, and Amazon Transcribe converts speech to text. Leila's recorded reminder calls could use Polly today; a conversational phone assistant would need something like Nova Sonic and much more careful design. The group parked that idea.

Not every model is in every Region

Model availability differs by AWS Region. Some models are offered only in certain Regions, and some are reachable from other Regions through cross-Region inference, which routes requests to wherever capacity is available within a defined set of Regions. For Cascade Ridge this is a real constraint. Its primary Region is US West (Oregon). Before choosing a model, Hannah wants to confirm which Regions a cross-Region inference profile can send requests to, so that the group can keep its promise that client information stays in the United States. The model catalog in the Bedrock console and the Bedrock documentation list current availability; Noor told Jonah never to rely on a blog post for it.

Other ways to classify models

Beyond modality, the exam expects you to recognize several other distinctions.

On AWS, the hosted path is Amazon Bedrock, which offers models from Amazon and other providers through one API without any servers to manage. The self-hosted path is Amazon SageMaker AI, where SageMaker JumpStart offers a catalog of open-weight and other models that you can deploy to endpoints in your own account, fine-tune, and run on instance types you choose. Lesson 8 compares them in detail.

# Matching each Cascade Ridge request to a model type by its modalities.
requests = [
    ("Summarize an overnight call note", "text", "text"),
    ("Find manual sections about controlled drugs", "text", "vector"),
    ("Draw a cartoon puppy for the newsletter", "text", "image"),
    ("Read the referring vet's name from a scanned letter", "image+text", "text"),
    ("Speak an appointment reminder", "text", "speech"),
]
model_type = {
    ("text", "text"): "large language model",
    ("text", "vector"): "embedding model",
    ("text", "image"): "image generation (diffusion) model",
    ("image+text", "text"): "multimodal model",
    ("text", "speech"): "speech model or text-to-speech service",
}
for task, inp, out in requests:
    print(f"{task:52} -> {model_type[(inp, out)]}")

Summarize an overnight call note                     -> large language model
Find manual sections about controlled drugs          -> embedding model
Draw a cartoon puppy for the newsletter              -> image generation (diffusion) model
Read the referring vet's name from a scanned letter  -> multimodal model
Speak an appointment reminder                        -> speech model or text-to-speech service

Leila's requests, sorted

By the end of the session Leila had her answer, and it was not one product but three. Newsletter illustrations need an image generation model, used under the labeling rules. Waiting-room captions are short text, which any capable language model can draft for a person to approve. Recorded reminder calls can use a text-to-speech service today. Jonah added a fourth line from the earlier sessions: the overnight call summaries need a language model, and the manual search needs an embedding model working with a language model.

Specialized and domain-adapted models

General models know a little about everything, including veterinary medicine, but their knowledge of a narrow field is uneven. Providers and research groups therefore release domain-adapted models trained further on specialist text, such as clinical literature, legal documents or source code, and code-specialized models are among the most widely used of these. A domain model may handle field-specific vocabulary and abbreviations better, but it may be weaker at general tasks, less often updated, or available only for self-hosting. For Cascade Ridge, Noor's advice was to test general models first on the group's own call notes, because a well-instructed general model with good context often matches a specialist one, and to consider a specialist model only if the tests showed a specific gap, such as repeated misreading of veterinary abbreviations.

Model cards and responsible choice

Every model in the Bedrock catalog has a model card, a short document from the provider describing the model's intended uses, its input and output modalities, its context window, the languages it supports, known limitations, and in many cases information about how it was trained and evaluated. Reading the model card is the first step in choosing a model, and it often answers questions that would otherwise take hours of testing: whether the model accepts PDF documents, whether it supports the languages a business needs, or whether the provider warns against certain uses. Jonah made reading the model card a fixed step in the group's selection checklist, alongside checking Region availability and running the group's own evaluation set. The responsible AI course in this program returns to model cards as a transparency tool.

One more practical point applies to every type of model in this lesson. A model's version matters as much as its name. Providers release new versions regularly, and an application that worked well on one version may behave differently on the next. The group therefore records the exact model identifier it tested, and treats a version change as a change that needs re-evaluation.