How generative AI and large language models work

On November 23 Maya arrives at the AI hour with a question her team asked over lunch: "How does it know what to say?" One agent had typed a returns question into a public chatbot at home and received a polite, sensible answer that cited a return period Harrow & Finch has never had. Another had asked it to write a birthday poem for her grandmother and been moved to tears. How can the same system be so good and so wrong? Arjun says the answer lies in what a large language model actually does, and that once Maya understands it, both results will make sense.

Neural networks in one page

Modern generative models are neural networks: large mathematical functions built from many simple units arranged in layers. Each unit takes numbers in, multiplies them by weights, adds them up and passes the result on. A network with many layers can represent extremely complicated relationships between its input and its output. The weights start random. Training adjusts them, a little at a time, so that the network's outputs get closer to the desired outputs across billions of examples. The adjustable weights are the model's parameters, and today's large models have billions or even hundreds of billions of them. Nobody writes the parameters by hand; they are the residue of training.

Neural networks work on numbers, so everything they handle must be turned into numbers first. For images that is natural, since a picture is already a grid of pixel values. For language, text is split into small pieces called tokens, and each token is mapped to a long list of numbers. Lesson 5 explains tokens and lesson 6 explains those lists of numbers, which are called embeddings.

The transformer

The breakthrough behind today's language models is an architecture called the transformer, described by researchers at Google in a 2017 paper titled "Attention Is All You Need". Its key idea is attention: when the model processes a word, it can weigh every other word in the passage and decide which ones matter for understanding it. In the sentence "The customer returned the drill because it was broken", attention lets the model connect "it" to "the drill" rather than to "the customer". Earlier designs read text one word at a time and struggled to keep track of long passages; transformers look at the whole passage at once and can be trained efficiently on huge amounts of text using many processors in parallel.

The GPT family of models, whose name stands for generative pre-trained transformer, along with most other modern language models, is built on this architecture. So are many image and speech models.

Next-token prediction

Here is the central idea. A large language model is trained to do one thing: given a sequence of tokens, predict the next one. During pre-training it reads an enormous amount of text, from books, websites, code and other sources, and at each position it guesses the next token, compares its guess with the real one and adjusts its parameters. Repeated across trillions of tokens, this simple task forces the model to absorb grammar, facts, styles of writing, patterns of reasoning and much more, because all of those help predict what comes next.

When you use the model, it generates text one token at a time. It reads your prompt, produces a probability for every token in its vocabulary, picks one, adds it to the text and repeats, until it produces a token that means "stop" or reaches a length limit. The answer to Maya's lunch question is now clear. When the agent asked about Harrow & Finch's return period, the model had never seen the company's policy, so it produced the most plausible-sounding number from all the return policies it had read. It was not lying; it was predicting. The poem was moving for the same reason: the model has read a great deal of poetry about grandmothers.

# A toy next-word predictor: count which word follows which in a tiny "training set".
from collections import Counter, defaultdict
text = ("unused items can be returned within 30 days . "
        "opened items can be exchanged within 30 days . "
        "unused items can be returned with a receipt .").split()
follows = defaultdict(Counter)
for a, b in zip(text, text[1:]):
    follows[a][b] += 1
word = "items"
sentence = [word]
for _ in range(6):
    nxt, n = follows[word].most_common(1)[0]
    sentence.append(nxt)
    word = nxt
print("prediction after 'can':", dict(follows["can"]))
print("generated:", " ".join(sentence))

prediction after 'can': {'be': 3}
generated: items can be returned within 30 days

The toy model counts which word follows which in three sentences and then always picks the most common next word. It produces fluent-looking text that mixes its sources, and it can only ever say things it has seen pieces of. A real language model is vastly more sophisticated, considering the whole context through attention rather than one previous word, but the principle of predicting the next piece from patterns is the same.

From a raw model to an assistant

A model fresh from pre-training is good at continuing text but not at following instructions. Ask it "What is the capital of Canada?" and it might continue with three more quiz questions. Developers therefore fine-tune it: they train it further on examples of instructions paired with good responses, so it learns to behave as an assistant. They then use human feedback, in which people rank several responses to the same prompt and the model is adjusted to prefer the kind of answers people rated highly. This stage, often called reinforcement learning from human feedback, also teaches the model to decline harmful requests and to be clearer and more honest about uncertainty. The result is the chat models available in Foundry.

Organizations can fine-tune a model further on their own examples to teach it a particular style or format, but Arjun explains that for most business needs this is not the first step. It is cheaper and more flexible to give the model the company's information in the prompt, a technique covered in lesson 7.

What models are good at and where they struggle

Because they are trained on so much text, large language models are strong at writing and rewriting, summarizing, translating, extracting information from text, answering questions about text they are given, classifying, writing and explaining code, and following multi-step instructions. They are weaker at precise arithmetic, at counting, at anything that requires information they were never given, at knowing what happened after their training data was collected, and at telling when they do not know something. Newer reasoning models, which spend extra computation working through a problem in steps before answering, are better at maths, logic and planning, but they are slower and cost more per request.

Choosing a model

The AI-901 outline asks candidates to identify an appropriate model based on its capabilities. Foundry offers a model catalog with models from Microsoft, OpenAI, Meta, Mistral, DeepSeek, Anthropic and others. Arjun shows Maya how to read a model's card, the page that describes what it does, and gives her a short checklist.

Foundry's catalog also offers models for other modalities: image generation models that create pictures from text, speech models, and embedding models that turn text into vectors for search. Choosing well is mostly about matching the model to the task, and then proving the choice with tests. A small, inexpensive model that answers 95 of Maya's 200 test questions in 100 correctly may be a better business choice than a large one that answers 97, if the large one costs ten times as much per answer.

# Illustrative cost and quality comparison on 1,000 chats a day (prices are made up for the example)
models = {
    "small model":  {"cost_per_chat": 0.002, "correct_in_100": 92},
    "medium model": {"cost_per_chat": 0.006, "correct_in_100": 96},
    "large model":  {"cost_per_chat": 0.030, "correct_in_100": 97},
}
chats = 1000
for name, m in models.items():
    daily = m["cost_per_chat"] * chats
    wrong = chats * (100 - m["correct_in_100"]) // 100
    print(f"{name:13s} daily cost {daily:6.2f}  wrong answers a day {wrong}")

small model   daily cost   2.00  wrong answers a day 80
medium model  daily cost   6.00  wrong answers a day 40
large model   daily cost  30.00  wrong answers a day 30

Maya likes the numbers but asks what happens to the wrong answers. That, Arjun says, is the real design question. If every uncertain answer is handed to a human agent, the cheaper model may be fine. If wrong answers go straight to customers, the extra accuracy may be worth paying for. The review group will decide after the tests in January.

Training data and its limits

Every model's knowledge stops at its training cutoff, the date after which no new text was included. A model trained on data up to early 2025 knows nothing about a policy Harrow & Finch changed in 2026, a product launched last week or today's weather, unless that information is supplied in the prompt or through a tool. Model cards in the Foundry catalog usually state the cutoff. The training data also shapes what the model is good at: a model that saw mostly English text will be weaker in other languages, and a model that saw little about a niche trade will make more mistakes about it. Arjun reminds Maya that hardware retail has its own vocabulary. A customer asking about a "two-by-four", a "GFCI outlet" or "PEX" expects the assistant to know what those mean, and testing must include such questions.

Training data also raises questions of rights and consent: who wrote the text, and were they asked? Several high-profile copyright cases about training data have been settled or decided since 2024, and the law is still developing in both Canada and the United States. For a company using models rather than training them, the practical points are to read the provider's terms, to rely on the provider's commitments about customer data, and to keep a person responsible for checking what is published. Microsoft states that prompts and responses sent to models deployed in Foundry are not used to train the shared foundation models, a point Carmen asks Arjun to confirm in writing for the review group.