Efficient inference and serving

Training a frontier model is a one-time expense measured in tens or hundreds of millions of dollars. Running it for millions of users is a continuing expense that, for a popular service, soon exceeds the cost of training. Every reduction in the memory, time and energy needed per token therefore matters, and inference efficiency has become one of the most active areas of engineering in AI. Lesson 2 showed how the KV cache and attention grow with context length, and Lesson 7 estimated the cost of an application from token prices. This lesson explains where those prices come from: the two phases of generation and their different bottlenecks, batching and memory management in serving systems, quantization, speculative decoding, distillation, parallelism across accelerators, and the cost and energy of inference.

Two phases of generation

Generating a response has two distinct phases. In the prefill phase, the model processes the whole prompt at once, computing keys and values for every prompt token in parallel. This is like a training forward pass: it involves large matrix multiplications and keeps the accelerator's arithmetic units busy, so it is compute-bound. In the decode phase, the model generates output tokens one at a time, each requiring a full pass through the network but processing only a single new token per sequence. For each step, every weight of the model must be read from memory, but each weight is used for only a few arithmetic operations, so decode is limited by memory bandwidth rather than arithmetic.

Users experience these phases through different measures. Time to first token depends mostly on prefill and grows with prompt length. Inter-token latency, or its inverse, output tokens per second per user, depends on decode. Operators care most about throughput, the total tokens processed per second across all users, which determines the cost per million tokens. These measures pull in different directions: serving each user alone gives the fastest response but the worst throughput, while packing many users together raises throughput at some cost in latency.

Memory bandwidth and batching

A simple model explains decode performance. At batch size one, each decode step reads all \(N\) parameters, taking time \(N b / B\), where \(b\) is the bytes per parameter and \(B\) the memory bandwidth. The arithmetic, about \(2N\) operations per token, takes far less time on modern accelerators. Because the weights are read once per step regardless of how many sequences are processed, serving many sequences in a batch spreads that cost: throughput rises almost in proportion to batch size until the arithmetic catches up with the memory reads and the system becomes compute-bound.

# Decode speed limits and speculative decoding
N = 70e9                         # parameters
bw = 3.35e12                     # memory bandwidth per accelerator, bytes/s
flops = 0.99e15                  # dense 16-bit throughput per accelerator, FLOP/s
gpus = 4                         # tensor parallel across 4 accelerators

for bytes_per in (2, 1, 0.5):                       # 16-bit, 8-bit, 4-bit weights
    t_mem = N * bytes_per / (bw * gpus)             # read all weights once per decode step
    for batch in (1, 32, 256):
        t_comp = 2 * N * batch / (flops * gpus * 0.5)   # assume 50% of peak in practice
        t = max(t_mem, t_comp)
        bound = "memory" if t_mem >= t_comp else "compute"
        print(f"{bytes_per * 8:2.0f}-bit, batch {batch:3d}: {1 / t:6.0f} tokens/s per sequence, "
              f"{batch / t:8.0f} tokens/s total ({bound}-bound)")

print()
# Speculative decoding: draft proposes g tokens, each accepted with probability a
def expected_tokens(a, g):
    return (1 - a ** (g + 1)) / (1 - a)

c = 0.05                          # cost of one draft step relative to one target step
for a in (0.6, 0.8, 0.9):
    for g in (3, 5):
        tok = expected_tokens(a, g)
        speedup = tok / (1 + g * c)
        print(f"acceptance {a:.1f}, draft length {g}: {tok:.2f} tokens per target pass, speedup {speedup:.2f}x")

16-bit, batch   1:     96 tokens/s per sequence,       96 tokens/s total (memory-bound)
16-bit, batch  32:     96 tokens/s per sequence,     3063 tokens/s total (memory-bound)
16-bit, batch 256:     55 tokens/s per sequence,    14143 tokens/s total (compute-bound)
 8-bit, batch   1:    191 tokens/s per sequence,      191 tokens/s total (memory-bound)
 8-bit, batch  32:    191 tokens/s per sequence,     6126 tokens/s total (memory-bound)
 8-bit, batch 256:     55 tokens/s per sequence,    14143 tokens/s total (compute-bound)
 4-bit, batch   1:    383 tokens/s per sequence,      383 tokens/s total (memory-bound)
 4-bit, batch  32:    383 tokens/s per sequence,    12251 tokens/s total (memory-bound)
 4-bit, batch 256:     55 tokens/s per sequence,    14143 tokens/s total (compute-bound)

acceptance 0.6, draft length 3: 2.18 tokens per target pass, speedup 1.89x
acceptance 0.6, draft length 5: 2.38 tokens per target pass, speedup 1.91x
acceptance 0.8, draft length 3: 2.95 tokens per target pass, speedup 2.57x
acceptance 0.8, draft length 5: 3.69 tokens per target pass, speedup 2.95x
acceptance 0.9, draft length 3: 3.44 tokens per target pass, speedup 2.99x
acceptance 0.9, draft length 5: 4.69 tokens per target pass, speedup 3.75x

The first part of the output shows the pattern. At batch size one in 16-bit precision, the model generates about 96 tokens per second; at batch size 32, each user still sees about 96 tokens per second, but total throughput is 32 times higher; at batch 256, the system is compute-bound, each user slows to about 55 tokens per second, and total throughput reaches about 14,000 tokens per second. Halving the bytes per weight doubles single-stream speed while the system is memory-bound. The model ignores the KV cache, which must also be read at every step and grows with batch size and context length (Lesson 2), so real systems become memory-bound again with long contexts, but the principle holds: batching is what makes language model serving affordable.

Serving systems use several techniques to batch efficiently:

• Continuous batching. Requests arrive at random times and produce outputs of different lengths. Rather than waiting for a whole batch to finish, the scheduler adds new requests and removes finished ones at every decode step, keeping the batch full.
• Paged KV cache memory. Allocating a contiguous block of cache memory for each request's maximum length wastes most of it. The PagedAttention technique, introduced with the open-source vLLM serving system in 2023, stores the cache in small fixed-size blocks allocated on demand, like virtual memory pages in an operating system, allowing far more requests to share the accelerator.
• Prefix caching. Requests that share a prefix, such as the same system prompt or document, reuse its cached keys and values instead of recomputing them; this is the basis of the prompt caching discounts offered by API providers.
• Disaggregated serving. Because prefill is compute-bound and decode memory-bound, some systems run them on separate pools of accelerators sized for each, transferring the KV cache between them.

Quantization

Quantization stores weights, and sometimes activations and the KV cache, in fewer bits. Training normally uses 16-bit floating point (bfloat16) for weights in the forward pass. Inference can use 8-bit integers or 8-bit floating point (FP8), halving memory and memory traffic, and 4-bit formats, quartering them. Recent accelerators support FP8 arithmetic directly, and the newest generation adds 4-bit floating point formats, so lower precision speeds up computation as well as memory access.

A quantized value is an integer multiplied by a scale factor. The choice of how many weights share one scale matters greatly, because language model weights and activations contain outliers, a few values much larger than the rest. With one scale for a whole matrix (per-tensor), the outliers set the scale and the ordinary small weights are represented by only a few levels. Per-channel scales (one per output row) and group-wise scales (one per group of, say, 64 or 128 weights) isolate the outliers.

import numpy as np

# Quantizing a weight matrix: per-tensor vs per-channel int8, and group-wise int4
rng = np.random.default_rng(0)
W = rng.normal(0, 0.02, size=(256, 1024))
W[7, :] *= 12                                  # one output channel with much larger weights (an outlier row)
x = rng.normal(0, 1, size=1024)
y = W @ x

def quantize(w, bits, axis=None, group=None):
    qmax = 2 ** (bits - 1) - 1
    if group:                                  # separate scale for each group of consecutive weights in a row
        g = w.reshape(w.shape[0], -1, group)
        s = np.abs(g).max(axis=2, keepdims=True) / qmax
        return (np.round(g / s).clip(-qmax, qmax) * s).reshape(w.shape)
    s = np.abs(w).max(axis=axis, keepdims=True) / qmax if axis is not None else np.abs(w).max() / qmax
    return np.round(w / s).clip(-qmax, qmax) * s

for name, Wq, bits in [("int8 per-tensor", quantize(W, 8), 8),
                       ("int8 per-channel", quantize(W, 8, axis=1), 8),
                       ("int4 per-channel", quantize(W, 4, axis=1), 4),
                       ("int4 groups of 64", quantize(W, 4, group=64), 4)]:
    err = np.linalg.norm(Wq @ x - y) / np.linalg.norm(y)
    print(f"{name:<18} relative output error {err:.4f}, memory {bits / 16:.0%} of 16-bit")

int8 per-tensor    relative output error 0.0818, memory 50% of 16-bit
int8 per-channel   relative output error 0.0081, memory 50% of 16-bit
int4 per-channel   relative output error 0.1596, memory 25% of 16-bit
int4 groups of 64  relative output error 0.1187, memory 25% of 16-bit

One outlier row inflates the per-tensor int8 error tenfold compared with per-channel scales. At 4 bits, simple rounding introduces substantial error even with group-wise scales, because only 15 levels are available. Practical 4-bit methods do better than naive rounding: GPTQ quantizes weights one at a time and adjusts the remaining weights to compensate for each rounding error, using a small calibration dataset, and AWQ identifies the weights that matter most for typical activations and scales them to preserve their precision. With such methods, 8-bit inference is usually indistinguishable from 16-bit in quality, and 4-bit weight quantization of large models typically costs only a small loss on benchmarks, though smaller models and demanding tasks such as long reasoning and mathematics are more sensitive. Every quantized model should be evaluated on the tasks it will serve, using the methods of Lesson 6.

Speculative decoding

Decode is slow because each step processes one token per sequence while the accelerator could process many. Speculative decoding exploits this spare capacity. A small, fast draft model proposes the next \(\gamma\) tokens. The large target model then processes all of them in a single pass, which costs about the same as generating one token because the step is memory-bound, and checks each proposed token in order. A token is accepted with a probability that depends on how the draft and target distributions compare; at the first rejection, a replacement token is sampled from a corrected distribution and the rest of the draft is discarded. The acceptance rule is designed so that the output has exactly the same distribution as sampling from the target model alone: speculative decoding makes generation faster without changing what is generated.

If each draft token is accepted with probability \(\alpha\), independently, the expected number of tokens produced per target pass is:

The second part of the code output above shows speedups of about 2 to nearly 4 times for typical acceptance rates, after allowing for the draft model's own cost. Acceptance is high for predictable text such as code and formulaic writing, and lower for creative text at high temperature. Variants avoid a separate draft model by adding extra prediction heads to the target model (as in Medusa and EAGLE) or by drafting from the model's own earlier layers, and some models are now trained to predict several tokens at once for this purpose.

Smaller models: distillation and pruning

Often the cheapest way to serve a task is a smaller model. Knowledge distillation trains a small student model to match the output probabilities of a large teacher, which conveys more information per example than the single correct token: the teacher's probabilities show which alternatives are nearly right. Distillation is widely used to produce the small and medium versions in a model family, and reasoning models have been distilled by fine-tuning smaller models on the teacher's reasoning traces, as DeepSeek did in early 2025. Pruning removes weights, attention heads or whole layers that contribute least, followed by brief retraining.

Many applications combine models of different sizes through routing: a small model handles simple requests, and only difficult ones go to a large model. Routing decisions can be made by a classifier, by the small model's own confidence, or by rules based on the type of request.

Mixture-of-experts models (Lesson 1) have their own serving trade-offs. Only a few experts are active per token, so the arithmetic per token is that of a much smaller model, but all experts must be held in memory, and the expert chosen for each token varies, so large deployments spread experts across many accelerators (expert parallelism) and must balance the load among them.

Parallelism and deployment

Large models do not fit on one accelerator. Tensor parallelism splits each layer's matrices across several accelerators in the same server, which communicate over fast links after every layer; pipeline parallelism assigns different layers to different servers. Serving platforms combine these with replication of the whole model to handle more users, load balancing across replicas, and autoscaling as demand rises and falls through the day. AI-480 covers the hardware and parallelism in depth, and the DCE courses cover the facilities.

At the other end of the scale, on-device inference runs models of a few billion parameters on phones and laptops, using 4-bit quantization and dedicated neural processing units. On-device models offer privacy (data does not leave the device), offline use and no per-token cost, at the price of lower capability, and many products combine them with larger cloud models for harder requests.

Cost and energy

The price of a given level of capability has fallen rapidly. Analyses of API prices have found that the cost of reaching a fixed benchmark score fell by roughly an order of magnitude or more per year between 2022 and 2025, through better models, quantization, batching and new hardware. At the same time, total spending on inference has grown, because cheaper tokens encourage more use, longer contexts, agents that run for many steps, and reasoning models that generate thousands of hidden tokens per answer.

Energy per query depends on the model, the hardware, the prompt and output lengths and the utilization of the data center. Published estimates vary widely. In August 2025, Google reported that the median text prompt to its Gemini apps used about 0.24 watt-hours, including idle capacity and data center overhead, while long reasoning queries and image or video generation use far more. Efficiency gains per query are real, but rising demand means that the total electricity used by AI inference continues to grow, which is a central concern of the data center engineering courses.