Fine-tuning and learning from human feedback

A pretrained base model knows a great deal but behaves like an autocomplete engine: asked a question, it may continue with a list of similar questions, produce something harmful because similar text appeared online, or make up an answer in a confident tone. Post-training is the set of techniques that turns a base model into a useful, honest and safe assistant: supervised fine-tuning on demonstrations, learning from human (or AI) preferences, and reinforcement learning on tasks with checkable answers. These stages use a small fraction of the compute of pretraining but have an outsized effect on how a model behaves. This lesson explains each technique, parameter-efficient methods that make fine-tuning affordable, and the pitfalls of optimizing against imperfect measures of what people want.

Supervised fine-tuning

Supervised fine-tuning (SFT), also called instruction tuning, continues training the base model on examples of the desired behavior: prompts paired with high-quality responses, written by people or by other models and reviewed. The loss is the same next-token cross-entropy as pretraining, usually computed only on the response tokens. A few thousand to a few million examples covering diverse tasks (answering questions, writing, coding, refusing harmful requests politely, following formats, holding multi-turn conversations) teach the model the format and style of an assistant. Research such as the 2023 LIMA study suggested that much of a model's knowledge and ability comes from pretraining, and that a relatively small set of carefully chosen examples can unlock it.

Conversations are formatted with a chat template that marks the roles (system, user, assistant) with special tokens, so the model learns where its turn starts and ends. A system prompt sets instructions that apply to the whole conversation.

SFT has limits. The model only imitates what demonstrators wrote; it cannot easily learn to be better than its demonstrations, and it does not learn from comparisons, such as which of two good answers is better. Writing ideal answers for hard tasks is also slow and expensive. Preference learning addresses these limits.

Reinforcement learning from human feedback

Reinforcement learning from human feedback (RLHF), popularized by OpenAI's InstructGPT work in 2022, proceeds in three steps:

1. Supervised fine-tuning, as above.

2. Reward modeling: the model generates several responses to each prompt; human raters compare them and choose the better one. A reward model (usually initialized from the language model, with a scalar output head) is trained to predict these preferences.

3. Reinforcement learning: the language model (the policy) generates responses, the reward model scores them, and an RL algorithm such as proximal policy optimization (PPO) updates the policy to increase reward, with a penalty for drifting too far from the SFT model.

Reward models are usually trained with the Bradley-Terry model of pairwise comparisons. If responses \(a\) and \(b\) have rewards \(r_a\) and \(r_b\), the probability that a rater prefers \(a\) is modeled as:

and the reward model is trained to maximize the likelihood of the observed choices, a logistic regression on reward differences. The RL objective then maximizes expected reward minus a Kullback-Leibler (KL) divergence penalty that keeps the policy close to the reference model:

The KL penalty matters because the reward model is only an approximation of human judgment. Without it, the policy finds responses that score highly but that people would not actually prefer, a failure called reward hacking or overoptimization.

import numpy as np

rng = np.random.default_rng(1)
w_true = np.array([2.0, 0.5, -0.3])


def responses(n):
    helpful = rng.normal(size=n)
    polite = rng.normal(size=n)
    length = rng.lognormal(0, 0.8, size=n)
    return np.column_stack([helpful, polite, length])


def label(xa, xb, bias):
    w = w_true + np.array([0, 0, bias])
    p = 1 / (1 + np.exp(-(xa - xb) @ w))
    return (rng.random(len(p)) < p).astype(float)


def fit(xa, xb, y, steps=3000, lr=0.1):
    w = np.zeros(3)
    X = xa - xb
    for _ in range(steps):
        p = 1 / (1 + np.exp(-X @ w))
        w -= lr * X.T @ (p - y) / len(y)
    return w


xa, xb = responses(4000), responses(4000)
for bias in [0.0, 0.8]:
    y = label(xa, xb, bias)
    w = fit(xa, xb, y)
    ta, tb = responses(2000), responses(2000)
    agree = np.mean(((ta - tb) @ w > 0) == ((ta - tb) @ w_true > 0))
    print(f"annotator length bias {bias}: learned w = {np.round(w, 2)}  agrees with true preference {agree:.2f}")
    for n in [1, 4, 16, 64, 256]:
        trues = []
        for _ in range(300):
            c = responses(n)
            trues.append(c[np.argmax(c @ w)] @ w_true)
        print(f"   best of {n:>3} by reward model: mean true reward {np.mean(trues):5.2f}")

annotator length bias 0.0: learned w = [ 2.08  0.52 -0.32]  agrees with true preference 1.00
   best of   1 by reward model: mean true reward -0.69
   best of   4 by reward model: mean true reward  1.82
   best of  16 by reward model: mean true reward  3.32
   best of  64 by reward model: mean true reward  4.46
   best of 256 by reward model: mean true reward  5.56
annotator length bias 0.8: learned w = [2.03 0.52 0.53]  agrees with true preference 0.89
   best of   1 by reward model: mean true reward -0.45
   best of   4 by reward model: mean true reward  1.48
   best of  16 by reward model: mean true reward  2.62
   best of  64 by reward model: mean true reward  3.37
   best of 256 by reward model: mean true reward  2.83

The simulation shows both how reward models work and how they fail. With unbiased annotators, the learned weights match the true preferences and choosing the best of more samples keeps improving true quality. When annotators slightly favor longer answers, the reward model learns to reward length; selecting harder against it first helps, then hurts, as the selection increasingly picks long but worse answers. Real RLHF has shown similar effects, such as models becoming verbose, sycophantic (agreeing with the user) or overconfident because raters tended to prefer such answers.

Direct preference optimization and other variants

PPO-based RLHF is complex: it trains and runs several models at once and is sensitive to hyperparameters. Direct preference optimization (DPO), introduced in 2023, showed that the KL-regularized RLHF objective has a closed-form optimal policy, which allows the policy to be trained directly on preference pairs with a simple classification-style loss, without a separate reward model or RL loop:

where \(y_w\) is the preferred response and \(y_l\) the rejected one. DPO and its many variants are widely used, especially in open-model development, because they are simpler and cheaper; large labs use a mix of online RL and offline preference methods.

Collecting human preferences at scale is expensive and can expose raters to disturbing content. Reinforcement learning from AI feedback (RLAIF) uses a model to make the comparisons, guided by written principles. Anthropic's Constitutional AI (2022) is an example: the model critiques and revises its own responses according to a set of principles (a "constitution") and AI-generated preferences train the reward model, reducing the need for human labels on harmful content while making the guiding values explicit.

Reinforcement learning with verifiable rewards and reasoning

For tasks with checkable answers, such as math problems with known solutions or code with unit tests, the reward does not need to come from a learned model. Reinforcement learning with verifiable rewards (RLVR) gives a reward when the final answer is correct or the tests pass. Training a model this way on many problems teaches it to produce long chains of reasoning that check intermediate steps, explore alternatives and correct mistakes, because such reasoning leads to correct answers more often. This is how reasoning models such as OpenAI's o-series and DeepSeek-R1 were developed; DeepSeek reported in January 2025 that applying large-scale RL with rule-based rewards to its base model produced strong reasoning behavior, including self-verification, with relatively simple methods such as group relative policy optimization (GRPO), which compares several sampled answers to the same problem instead of training a separate value model.

Verifiable rewards can still be gamed: a coding model may learn to special-case the tests rather than solve the problem, so test suites, environments and monitoring must be designed carefully.

Parameter-efficient fine-tuning

Fully fine-tuning a large model updates all its parameters and requires the same memory-hungry optimizer state as pretraining. Parameter-efficient fine-tuning (PEFT) methods update only a small number of parameters. The most popular is low-rank adaptation (LoRA, 2021), which freezes the original weight matrix \(W\) and learns a low-rank update:

with rank \(r\) much smaller than \(d\) (often 8 to 64). This works because the weight changes needed to adapt a model to a task tend to have low intrinsic rank. LoRA adapters are small files that can be swapped for different tasks on top of one base model, and QLoRA combines LoRA with a 4-bit quantized base model so that large models can be fine-tuned on a single GPU.

import numpy as np

rng = np.random.default_rng(0)
d = 512
U = rng.normal(size=(d, 4))
V = rng.normal(size=(4, d))
delta = U @ V / d ** 0.5 + 0.01 * rng.normal(size=(d, d))

u, s, vt = np.linalg.svd(delta)
total = np.sum(s ** 2)
print("full update parameters:", d * d)
for r in [1, 2, 4, 8, 16]:
    approx = (u[:, :r] * s[:r]) @ vt[:r]
    err = np.linalg.norm(delta - approx) / np.linalg.norm(delta)
    print(f"rank {r:>2}: LoRA params {2 * d * r:>6} ({200 * r / d:5.2f}%)  energy kept {np.sum(s[:r] ** 2) / total:.3f}  rel. error {err:.3f}")

layers, d_model, r = 32, 4096, 16
per_matrix = 2 * d_model * r
lora = layers * 4 * per_matrix
print(f"LoRA r={r} on 4 attention matrices x {layers} layers: {lora / 1e6:.1f} M trainable parameters")

full update parameters: 262144
rank  1: LoRA params   1024 ( 0.39%)  energy kept 0.290  rel. error 0.843
rank  2: LoRA params   2048 ( 0.78%)  energy kept 0.549  rel. error 0.671
rank  4: LoRA params   4096 ( 1.56%)  energy kept 0.987  rel. error 0.112
rank  8: LoRA params   8192 ( 3.12%)  energy kept 0.988  rel. error 0.111
rank 16: LoRA params  16384 ( 6.25%)  energy kept 0.988  rel. error 0.107
LoRA r=16 on 4 attention matrices x 32 layers: 16.8 M trainable parameters

Quantization is used for inference as well: storing weights in 8-bit or 4-bit formats instead of 16-bit cuts memory and often speeds up generation, with a modest loss of accuracy that grows at very low precision.

Alignment, safety and evaluation

Post-training is where developers shape a model's values and behavior: being helpful while declining genuinely harmful requests, being honest about uncertainty, resisting manipulation, and treating users fairly. This is hard to get right. Models can be over-cautious (refusing harmless requests), sycophantic, or inconsistent, and adversarial prompts (jailbreaks) can sometimes bypass training. Developers evaluate models with benchmarks, human evaluations, automated red teaming and specialized tests for dangerous capabilities (for example, in cybersecurity or biology), and publish system cards and safety frameworks describing what they test and what safeguards they apply. These practices are increasingly expected by regulators and customers.

Lesson 6 explains how the effects of fine-tuning and preference training are measured, and the statistical care needed to compare models fairly.