Large language models: alignment, prompting and decoding

A pretrained language model (Lesson 7) is a powerful text predictor, but it is not yet a useful assistant. Asked a question, it may continue with more questions, imitate a forum thread, or produce fluent text that is false or harmful, because its only goal is to continue text the way its training data would. This lesson explains how pretrained models are turned into assistants through instruction tuning and preference optimization, how people get good results from them through prompting, in-context learning and step-by-step reasoning, how the decoding settings control the text they generate, and why they sometimes produce confident but false statements, known as hallucinations.

From pretrained model to assistant

Modern assistants are built in stages, collectively called post-training:

• Supervised fine-tuning (SFT), also called instruction tuning: the pretrained model is fine-tuned on examples of instructions paired with high-quality responses, written by people or generated and filtered. Studies such as FLAN showed that training on many tasks phrased as instructions greatly improves zero-shot performance on new tasks.
• Preference optimization: people (or AI systems guided by written principles) compare pairs of model responses and choose the better one. The model is then trained to prefer responses like the chosen ones, improving helpfulness, honesty and harmlessness in ways that are hard to capture with demonstrations alone.
• Reasoning and tool-use training: models are further trained, often with reinforcement learning on problems with checkable answers such as mathematics and code, to reason step by step and to call tools such as search engines and code interpreters (Lesson 11).

Reinforcement learning from human feedback

Reinforcement learning from human feedback (RLHF), covered from the reinforcement learning side in AI-330, Lesson 12, has three steps. First, collect comparisons: for many prompts, sample two or more responses and have labelers rank them. Second, train a reward model, usually a copy of the language model with a scalar output, so that the preferred response scores higher. The Bradley-Terry model gives the probability that response \(y_w\) is preferred to \(y_l\) as

and the reward model is trained to maximize the log of this probability over the comparisons. Third, optimize the language model (the policy) with reinforcement learning, typically proximal policy optimization (PPO), to maximize the reward while a penalty on the KL divergence from the SFT model keeps it from drifting into strange text that exploits weaknesses of the reward model:

Without that penalty, models learn to game the reward model, a form of reward hacking: overly long answers, flattery, or confident tone, which reward models trained on human preferences tend to favor.

Direct preference optimization

Direct preference optimization (DPO), introduced in 2023, reaches the same objective without training a separate reward model or running reinforcement learning. It shows that the optimal policy for the KL-penalized objective implies a reward equal to \(\beta \log \frac{\pi(y \mid x)}{\pi_{\mathrm{SFT}}(y \mid x)}\), so the policy can be trained directly on comparisons with a simple classification-like loss:

DPO and its variants are simpler and more stable than PPO, and are widely used, particularly for open models. Other approaches replace some human labels with AI feedback: in constitutional AI, a model critiques and revises responses according to a written set of principles, and AI-generated preferences train the reward model.

Prompting

Because an assistant follows instructions written in natural language, the prompt is the main interface for specifying a task. Good prompts share the qualities of good instructions to a capable colleague who knows nothing about the situation:

• State the task, audience and purpose clearly, and give the relevant context and constraints.
• Specify the format of the output: length, structure, headings, or a schema such as JSON.
• Provide examples of desired input and output (few-shot prompting), choosing varied, representative examples.
• Separate instructions from data with clear delimiters such as XML tags, especially when the data come from users or documents.
• Use a system prompt for persistent instructions such as role, tone and policies.
• Ask the model to think through the problem before answering when the task needs reasoning.

Prompts should be developed like code: written against a set of test cases, evaluated, and revised, rather than tuned on one example. Small changes in wording or example order can change results, so conclusions drawn from a single prompt are fragile.

In-context learning

In-context learning, discovered with GPT-3, lets a model perform a new task from a few examples in the prompt without any weight update. How it works is still studied: research suggests that transformers can implement simple learning algorithms in their forward pass, using attention to match the new input with the examples, and that the examples often serve mostly to identify the task and output format rather than to teach new knowledge. Zero-shot prompting gives instructions only; one-shot and few-shot prompting add examples. For instruction-tuned models, clear instructions often matter more than examples, but examples remain valuable for specifying formats and subtle distinctions.

Chain-of-thought reasoning

Chain-of-thought prompting asks the model to write out intermediate reasoning steps before the final answer, either by showing worked examples or simply by adding an instruction such as "think step by step". Because each generated token is computed with a fixed amount of computation, generating intermediate steps gives the model more computation and a written scratchpad for multi-step problems, and it substantially improves accuracy on arithmetic, logic and planning tasks for sufficiently capable models. Reasoning models are trained with reinforcement learning to produce long chains of thought, exploring and checking alternatives before answering, and their accuracy improves as they are allowed to think longer, a form of inference-time scaling.

A simple way to use more inference compute is self-consistency: sample several independent reasoning chains at a nonzero temperature and take a majority vote over their final answers.

import numpy as np

# Self-consistency: sample several reasoning chains and take a majority vote on the final answer.
# Simulate a model whose single chain is correct with probability p; wrong answers spread over 4 alternatives.
rng = np.random.default_rng(1)
trials = 20000

def vote_accuracy(p, k):
    correct = rng.random((trials, k)) < p
    wrong = rng.integers(1, 5, size=(trials, k))
    answers = np.where(correct, 0, wrong)
    counts = np.stack([(answers == a).sum(axis=1) for a in range(5)], axis=1)
    counts = counts + rng.random(counts.shape) * 0.1            # break ties at random
    return np.mean(counts.argmax(axis=1) == 0)

for p in (0.3, 0.5, 0.7):
    row = ", ".join(f"k={k}: {vote_accuracy(p, k):.3f}" for k in (1, 5, 11, 21))
    print(f"single-chain accuracy {p}: {row}")
# When wrong answers concentrate on one popular mistake, voting helps less
def vote_concentrated(p, k):
    correct = rng.random((trials, k)) < p
    wrong = np.where(rng.random((trials, k)) < 0.8, 1, rng.integers(2, 5, size=(trials, k)))
    answers = np.where(correct, 0, wrong)
    counts = np.stack([(answers == a).sum(axis=1) for a in range(5)], axis=1)
    counts = counts + rng.random(counts.shape) * 0.1
    return np.mean(counts.argmax(axis=1) == 0)

print("single-chain accuracy 0.4, 80% of errors on one answer: " + ", ".join(f"k={k}: {vote_concentrated(0.4, k):.3f}" for k in (1, 5, 11, 21)))

single-chain accuracy 0.3: k=1: 0.303, k=5: 0.378, k=11: 0.458, k=21: 0.564
single-chain accuracy 0.5: k=1: 0.504, k=5: 0.711, k=11: 0.883, k=21: 0.971
single-chain accuracy 0.7: k=1: 0.695, k=5: 0.924, k=11: 0.993, k=21: 1.000
single-chain accuracy 0.4, 80% of errors on one answer: k=1: 0.399, k=5: 0.409, k=11: 0.382, k=21: 0.350

When a single chain is right more often than any particular wrong answer, voting amplifies that edge: at 50% single-chain accuracy, 21 votes reach 97%. Voting cannot fix systematic errors, however: if most mistakes land on the same wrong answer, which happens when a problem contains a tempting trap, voting can make accuracy worse. Verifiers that check answers, such as running code against tests, are more reliable where available.

Decoding: from probabilities to text

At each step a language model outputs a probability distribution over the vocabulary, and a decoding strategy chooses the next token. Greedy decoding and beam search (Lessons 4 and 5) pick the most likely tokens, which suits tasks with one correct answer but tends to produce bland, repetitive text in open-ended generation. Sampling picks tokens at random according to the distribution, usually modified by these controls:

• Temperature \(T\), introduced with sampling in Lesson 4, divides the logits before the softmax, \(p_i \propto \exp(z_i / T)\). Values below 1 sharpen the distribution toward the most likely tokens; values above 1 flatten it, giving more varied and more error-prone text. As \(T \to 0\), sampling becomes greedy.
• Top-k sampling keeps only the \(k\) most likely tokens and renormalizes.
• Top-p (nucleus) sampling keeps the smallest set of tokens whose cumulative probability exceeds \(p\), so the number of candidates adapts to how confident the model is.
• Repetition and frequency penalties reduce the probability of tokens already used.

import numpy as np

# Next-token logits from a language model after the prompt "The capital of Canada is"
tokens = ["Ottawa", "Toronto", "Montreal", "a", "the", "located", "Vancouver", "not"]
logits = np.array([6.0, 3.5, 3.0, 2.8, 2.5, 2.2, 1.5, 1.0])

def probs(z, T):
    p = np.exp((z - z.max()) / T)
    return p / p.sum()

for T in (0.5, 1.0, 1.5):
    p = probs(logits, T)
    H = -(p * np.log2(p)).sum()
    top = ", ".join(f"{t} {q:.3f}" for t, q in zip(tokens[:3], p[:3]))
    print(f"temperature {T}: {top}; entropy {H:.2f} bits")

def top_p(p, P):
    order = np.argsort(-p)
    keep = order[: np.searchsorted(np.cumsum(p[order]), P) + 1]
    q = np.zeros_like(p)
    q[keep] = p[keep]
    return q / q.sum(), keep

p = probs(logits, 1.5)
for P in (0.5, 0.9):
    q, keep = top_p(p, P)
    word = "token" if len(keep) == 1 else "tokens"
    print(f"top-p {P} at temperature 1.5 keeps {len(keep)} {word}: " + ", ".join(tokens[i] for i in keep))
# How often does sampling give a wrong answer?
rng = np.random.default_rng(0)
for T in (0.5, 1.0, 1.5):
    draws = rng.choice(len(tokens), size=10000, p=probs(logits, T))
    print(f"temperature {T}: Ottawa sampled {np.mean(draws == 0):.1%} of the time")

temperature 0.5: Ottawa 0.988, Toronto 0.007, Montreal 0.002; entropy 0.12 bits
temperature 1.0: Ottawa 0.804, Toronto 0.066, Montreal 0.040; entropy 1.20 bits
temperature 1.5: Ottawa 0.587, Toronto 0.111, Montreal 0.079; entropy 2.07 bits
top-p 0.5 at temperature 1.5 keeps 1 token: Ottawa
top-p 0.9 at temperature 1.5 keeps 5 tokens: Ottawa, Toronto, Montreal, a, the
temperature 0.5: Ottawa sampled 98.7% of the time
temperature 1.0: Ottawa sampled 80.4% of the time
temperature 1.5: Ottawa sampled 59.2% of the time

At temperature 0.5 the model answers Ottawa almost every time; at 1.5 it does so only about 59% of the time, otherwise starting with a different city or with words that may still lead to a correct sentence ("located in Ottawa"). Top-p 0.9 removes the long tail of unlikely tokens such as "Vancouver" and "not". For factual answers, extraction and code, use a low temperature; for brainstorming and creative writing, a moderate temperature with top-p around 0.9 to 0.95 is a common starting point.

Structured output modes constrain decoding so that only tokens consistent with a grammar or JSON schema can be chosen, guaranteeing parseable output for programs that consume it.

Hallucination

Hallucination is the generation of fluent, confident content that is false or unsupported: invented citations, wrong dates, nonexistent functions in code, or summaries that add facts absent from the source. It arises from how models are trained. Pretraining rewards plausible continuations, not truth; facts seen rarely in training are stored imperfectly; and models are often rewarded for answering rather than for saying they do not know. Hallucination is especially likely for obscure topics, precise numbers and quotations, recent events after the training cutoff, and long outputs where errors compound.

Mitigations reduce but do not eliminate it:

• Ground the model in sources: supply relevant documents in the prompt or through retrieval (Lesson 9), and ask it to answer only from them and to cite passages.
• Allow and encourage abstention: instruct the model that "I do not know" is acceptable, and train models to calibrate their confidence.
• Use tools for facts and computation: search, databases and code execution (Lesson 11).
• Verify: check citations exist and support the claims, run generated code, and have a second model or a person review high-stakes outputs.
• Evaluate factuality systematically with benchmarks and spot checks (Lesson 12).

Safety and misuse

Assistants are trained to refuse clearly harmful requests and to be careful in sensitive domains, but they remain vulnerable to jailbreaks, prompts crafted to bypass safety training, and to prompt injection, in which instructions hidden in documents, web pages or emails processed by the model hijack its behavior. Prompt injection is a serious risk for systems that read untrusted content and can take actions (Lesson 11): treat model outputs as untrusted, limit permissions, and require confirmation for consequential actions. Other concerns include bias inherited from training data, sycophancy (telling users what they want to hear), privacy leakage of memorized training data, and overreliance by users.