Transformers and large language models

"How does it know what to write next?" Jonah asked at the start of the third AI hour. He had spent the weekend trying a chatbot on old, anonymized call notes from the emergency hospital, and the summaries were better than he expected. One of them correctly noted that a dog had eaten raisins "about two hours before the call" even though the caller had said "around nine" and the call was logged at eleven. "It did arithmetic I didn't ask for."

Noor drew a row of boxes on the whiteboard. "It did not exactly do arithmetic. It predicted text that fits. But to see why that works so well, you need to know what is inside a large language model." This lesson opens the box: the transformer architecture behind nearly every modern large language model, the idea of attention, how a model generates text one token at a time, and the settings that control that process. You will not need to build a transformer for the exam, but you will need to explain what one does and why it matters for cost, speed and reliability.

Neural networks in one paragraph

A neural network is a mathematical function built from many simple units arranged in layers. Each unit takes numbers in, multiplies them by learned values called weights or parameters, adds them up and passes the result on. Training adjusts the parameters, a tiny step at a time, so the network's outputs get closer to the right answers on the training data. A large language model is a neural network with billions of parameters, trained on a huge amount of text. The size is where the word large comes from.

The transformer

Before 2017, language models usually read text one word after another, which made them slow to train and forgetful over long passages. That year a research paper titled "Attention Is All You Need" introduced the transformer, an architecture that looks at all the tokens in a passage at once and learns which ones matter to each other. Transformers could be trained on far more data, far faster, using many processors in parallel. Almost every large language model since, including the models available in Amazon Bedrock, is built on transformers.

A transformer used for generation works in four stages.

• Embedding. Each input token is turned into a vector, as in lesson 2, and information about its position in the sequence is added, because word order matters: "dog bit man" and "man bit dog" use the same tokens.
• Attention layers. The vectors pass through a stack of layers. In each one, every token gathers information from the other tokens it is related to. This is the step that lets the model connect "it" to "the dog" three sentences earlier.
• Feed-forward layers. Inside each layer, the gathered information is transformed further. These layers hold much of what the model learned during training about facts, grammar and style.
• Next-token scores. At the end, the model produces a score for every token in its vocabulary: how likely each one is to come next.

Many current models stack dozens of these layers. Some, called mixture-of-experts models, route each token through only part of the network to save computation. The exam does not test these internals; it tests the idea that a transformer reads the whole context and predicts the next token.

Attention

Self-attention is the transformer's central idea. For each token, the model computes how much it should draw on every other token in the context. In "The cat ate its food", the token "its" attends strongly to "cat", because that is what "its" refers to. The figure shows made-up attention weights for that sentence; real models compute many such patterns in parallel, called attention heads, each picking up a different kind of relationship.

Attention explains why the model in Jonah's experiment could connect "around nine" with the call time: both were in the context, and the model had learned during training that summaries of such calls usually state elapsed time. It also explains a cost. Attention compares tokens with each other, so the work grows quickly as the context gets longer. Very long inputs are slower and more expensive to process, which is one more reason to send a model only the chunks it needs.

How a large language model generates text

A language model writes by next-token prediction. Given the tokens so far, it produces scores for every possible next token. The scores are turned into probabilities, one token is chosen, it is appended to the sequence, and the whole process repeats until the model produces a special end token or reaches a length limit. A 200-token summary is the result of 200 rounds of this loop.

Two consequences follow. First, generation is sequential: the model cannot write the tenth word before the ninth, so output length drives response time. Second, the model never "decides" what to say as a whole. It commits to one token at a time, which is why it can start a sentence confidently and finish it with an invented detail.

Choosing the next token: temperature, top-k and top-p

How the model picks from the probabilities is controlled by inference parameters that you set on each request. They do not change the model; they change how its output is sampled.

• Temperature adjusts how adventurous the choice is. A low temperature, near 0, makes the model favour the most likely token almost every time, giving focused, repeatable output. A higher temperature spreads the choice across more tokens, giving more varied and creative output, with more risk of wandering off the facts.
• Top-k limits the choice to the k most likely tokens. With top-k of 5, only the five highest-scoring tokens are ever considered.
• Top-p, or nucleus sampling, limits the choice to the smallest set of tokens whose probabilities add up to p. With top-p of 0.9, the model chooses among the tokens that together make up 90 hundredths of the probability.
• Maximum length caps the number of output tokens, which also caps the cost and time of a response.
• Stop sequences are strings that end generation when the model produces them, such as a closing marker at the end of a form.

For the overnight call summaries, Noor recommended a low temperature: the group wants the same facts stated plainly every time, not creative variation. For drafting newsletter headlines, a higher temperature would give the marketing coordinator more options to choose from.

import math, random
random.seed(1)
# Made-up scores for the next word after "Advised owner to bring the dog in ..."
scores = {"tonight": 3.2, "immediately": 2.9, "tomorrow": 1.4, "eventually": 0.2, "dancing": -2.0}

def probs(scores, temperature):
    exps = {w: math.exp(s / temperature) for w, s in scores.items()}
    total = sum(exps.values())
    return {w: e / total for w, e in exps.items()}

for t in (0.2, 1.0, 2.0):
    p = probs(scores, t)
    picks = [random.choices(list(p), weights=list(p.values()))[0] for _ in range(20)]
    top = ", ".join(f"{w} {p[w]:.2f}" for w in sorted(p, key=p.get, reverse=True)[:3])
    print(f"temperature {t}: {top} | sample: {sorted(set(picks))}")

temperature 0.2: tonight 0.82, immediately 0.18, tomorrow 0.00 | sample: ['immediately', 'tonight']
temperature 1.0: tonight 0.51, immediately 0.38, tomorrow 0.08 | sample: ['immediately', 'tomorrow', 'tonight']
temperature 2.0: tonight 0.39, immediately 0.34, tomorrow 0.16 | sample: ['dancing', 'eventually', 'immediately', 'tomorrow', 'tonight']

The output shows the effect directly. At a low temperature almost all the probability sits on "tonight" and "immediately", and every sample is one of them. At a high temperature even "dancing" gets a share, and nonsense becomes possible. Jonah wrote one line in his notes: low temperature for facts, higher for ideas.

Nondeterminism

Because generation involves sampling, the same prompt can produce different outputs on different requests. This is called nondeterminism, and the exam lists it as a disadvantage of generative AI for some uses. Setting temperature to zero makes output much more consistent, but even then, small differences can occur because of how computations are distributed across hardware. Applications that need identical output every time, such as a legal disclaimer, should store the approved text and display it, not ask a model to regenerate it.

What training gives a model, and what it does not

Large language models are trained in stages. In pre-training, the model reads an enormous collection of text and learns to predict the next token; this is where it picks up grammar, facts, reasoning patterns and writing styles. In later stages, often called instruction tuning and alignment, it is trained further on examples of helpful answers and on human or automated feedback, so that it follows instructions, refuses clearly harmful requests and answers in a useful format. Lesson 7 places these stages in the foundation model lifecycle.

Training ends at a date called the knowledge cutoff. The model knows nothing that happened afterwards, and it knows nothing about Cascade Ridge's own records at all unless that information is supplied in the prompt or the model is customized. When Jonah asked a chatbot what the group's after-hours fee was, it gave a plausible number that was simply invented. The model had never seen the fee schedule.

Training also does not give the model a reliable sense of its own uncertainty. It produces the most fitting continuation whether or not it "knows" the answer, which is the root of hallucination, covered in lesson 6.

Context windows and long inputs

Every transformer has a context window, the maximum number of tokens it can attend to at once, counting the prompt and the response together. Current models offer windows from tens of thousands to around a million tokens, which sounds limitless until you multiply: a year of overnight call notes for the emergency hospital would still not fit in most of them, and sending even a fraction would be slow and expensive. Research and practical testing also show that models make better use of information near the start and end of a long input than in the middle, a pattern sometimes described as being "lost in the middle".

Two practical rules follow. First, a large context window is a ceiling, not a target: send what the task needs, not everything available. Second, put the most important material where the model will use it best, usually the instructions first and the question last, with the source material in between and clearly labelled. Jonah tried both orders on the same twenty test notes and found that labelling the call note with a clear heading and placing the instructions before it reduced omissions noticeably, a small result that convinced him context design was worth the effort.

Why the model seems to reason

Jonah's raisin example, where the model worked out an elapsed time it was not asked for, looks like reasoning. In a sense it is: during training on enormous amounts of text, the model learned patterns that include step-by-step explanations, calculations and logical arguments, and it can reproduce those patterns on new problems. Many current models are also trained to work through a problem in intermediate steps before answering, which improves results on multi-step tasks. But the reasoning is produced the same way as everything else, one likely token at a time, so it can contain mistakes that look perfectly confident. For anything numeric that matters, the safe pattern is the one from lesson 1: let a deterministic tool do the arithmetic and have the model explain or present the result.