Tokens, context windows and model parameters

Dana Leclair sends Jules a pricing page and one question: "What is a token, and why are we paying per million of them?" It is a fair question, and the answer turns out to explain much more than the bill. Tokens decide how much a model can read at once, how long its answers can be, how fast it responds and what it costs. This lesson covers tokens, context windows, the two meanings of the word parameter, and the settings that control how a model chooses its words.

Tokens

Language models do not read letters or whole words. They read tokens: chunks of text produced by a tokenizer, the component that splits input into pieces the model knows. A token is often a whole common word, part of a longer word, a punctuation mark or a space attached to a word. "Waterproof" might be one token or two; "Birchline" is probably split into several, because it is rare. Numbers, code and text in other languages often split into more tokens than everyday English.

Every token maps to an ID number in the model's vocabulary, which might contain a few hundred thousand entries. The model works entirely with these IDs and the patterns it learned among them. When it writes an answer, it produces one token at a time, each chosen based on everything before it. That is why responses appear word by word in a chat window: the model is literally generating them in sequence.

Images, audio and video are tokenized too. An image is divided into patches, and audio and video into short segments, each represented as tokens the model can process alongside text. The exact counts vary by model and settings, and providers document them.

A rough rule for English is that a token averages about three-quarters of a word, or equivalently about 1.3 tokens per word. Ravi warns Jules that this is only a planning rule. The real count depends on the tokenizer, and most model providers offer a token-counting tool or API so teams can measure exactly.

# A toy tokenizer: split text into known pieces, falling back to single letters.
vocab = ["water", "proof", "rain", "jacket", " ", "s", "the", "light", "weight", ".", ","]
vocab.sort(key=len, reverse=True)        # try longer pieces first

def tokenize(text):
    tokens, i = [], 0
    while i < len(text):
        for piece in vocab:
            if text.startswith(piece, i):
                tokens.append(piece); i += len(piece); break
        else:
            tokens.append(text[i]); i += 1   # unknown character becomes its own token
    return tokens

for phrase in ["waterproof rain jackets", "lightweight", "birchline"]:
    t = tokenize(phrase)
    print(len(t), "tokens:", t)

7 tokens: ['water', 'proof', ' ', 'rain', ' ', 'jacket', 's']
2 tokens: ['light', 'weight']
9 tokens: ['b', 'i', 'r', 'c', 'h', 'l', 'i', 'n', 'e']

The toy vocabulary is tiny, so the unfamiliar word "birchline" breaks into single letters, which is exactly why rare names and unusual words cost more tokens in real models. Real tokenizers learn their vocabularies from data, so common words become single tokens and rare ones split into a few pieces rather than letters.

Why tokens matter for cost and speed

Most generative AI services charge by the token, usually with separate prices for input tokens (everything sent to the model: instructions, documents, conversation history, images) and output tokens (everything the model writes back). Output tokens usually cost more, because generating them takes more computing work. Larger, more capable models cost more per token than smaller ones.

Speed follows tokens too. The time to the first token depends on how much input the model must process, and the time to finish depends on how many output tokens it writes. A shopping assistant that pastes the whole catalogue into every request will be slow and expensive; one that sends only the relevant products will be fast and cheap.

# Estimate monthly cost for two Birchline workloads. Prices are illustrative
# planning figures in US dollars per million tokens, not published rates.
PRICE = {"large": {"in": 1.25, "out": 10.00}, "small": {"in": 0.10, "out": 0.40}}

def monthly(model, requests, tokens_in, tokens_out):
    p = PRICE[model]
    return requests * (tokens_in * p["in"] + tokens_out * p["out"]) / 1_000_000

email = monthly("small", 20_000, 600, 20)       # classify emails into queues
descr = monthly("large", 300, 1_500, 400)       # draft product descriptions
chat_large = monthly("large", 90_000, 2_500, 250)
chat_small = monthly("small", 90_000, 2_500, 250)
print("email sorting, small model:", round(email, 2))
print("descriptions, large model:", round(descr, 2))
print("assistant, large vs small:", round(chat_large, 2), "vs", round(chat_small, 2))

email sorting, small model: 1.36
descriptions, large model: 1.76
assistant, large vs small: 506.25 vs 31.5

With these illustrative prices the email sorting and descriptions cost little, and the shopping assistant is the line to watch because of its volume. Dana circles the comparison between a large and a small model for the assistant: it is the kind of choice evaluation should settle, by testing whether the small model is good enough before paying for the large one.

Context windows

The context window is the maximum number of tokens a model can consider in a single request, counting the input and, for most models, the output it generates. Everything the model knows about the current task must fit inside it: the instructions, any documents or examples, the conversation so far, images and the answer.

Context windows have grown dramatically, from a few thousand tokens in early models to hundreds of thousands and, for some Gemini models, a million tokens or more. A million tokens is roughly the length of several long novels, a large codebase or hours of audio. Large windows make it possible to give a model an entire manual or a long contract in one request.

A large window is not a reason to send everything. Longer inputs cost more and respond more slowly, and models can pay less attention to details buried in the middle of very long inputs. Good design still selects what is relevant. Lesson 6 introduces retrieval-augmented generation, which finds the most relevant passages and sends only those.

When a conversation exceeds the window, something must give. Applications typically drop or summarize the oldest turns. Models do not remember previous conversations on their own; any memory an assistant appears to have is the application sending earlier messages, or saved notes, back in each new request.

Two meanings of parameters

Jules notices that "parameters" is used in two different ways, and Ravi confirms that the exam uses both.

Model parameters (also called weights) are the internal numbers a model learns during training. A model with billions of parameters has billions of adjustable values that together encode what it learned. More parameters generally allow more capability but require more memory and computing power to run, which is why smaller models are cheaper and faster. Customers never set these directly; training and tuning change them.

Generation parameters (also called sampling settings or inference parameters) are settings the caller chooses for each request to control how the model produces its output. They do not change the model; they change how it picks tokens. The main ones are temperature, top-p, top-k, maximum output tokens, stop sequences and safety settings.

How a model chooses each token

At every step, a model computes a score for every token in its vocabulary, converts the scores into probabilities and then picks one token. Always picking the most probable token gives predictable but sometimes repetitive text. Sampling from the probabilities gives more varied text. Generation parameters control that choice.

Temperature reshapes the probabilities before sampling. A low temperature, near zero, sharpens them so the most likely tokens dominate: output becomes focused, consistent and close to deterministic. A higher temperature flattens them so less likely tokens get more chances: output becomes more varied and creative, and also more likely to wander or make mistakes.

import math
scores = {"durable": 2.0, "rugged": 1.5, "lightweight": 1.0, "sparkly": -1.0}

def probabilities(scores, temperature):
    exp = {w: math.exp(s / temperature) for w, s in scores.items()}
    total = sum(exp.values())
    return {w: round(v / total, 3) for w, v in exp.items()}

for t in (0.2, 1.0, 2.0):
    print("temperature", t, probabilities(scores, t))

temperature 0.2 {'durable': 0.918, 'rugged': 0.075, 'lightweight': 0.006, 'sparkly': 0.0}
temperature 1.0 {'durable': 0.494, 'rugged': 0.3, 'lightweight': 0.182, 'sparkly': 0.025}
temperature 2.0 {'durable': 0.383, 'rugged': 0.299, 'lightweight': 0.233, 'sparkly': 0.086}

At temperature 0.2 nearly all the probability goes to "durable", the top choice. At 2.0 even "sparkly", an odd word for hiking boots, gets a real chance. That is the whole intuition: low temperature for facts, extraction and classification; moderate temperature for natural writing; higher temperature for brainstorming.

Top-k limits sampling to the k most likely tokens; with top-k set to 3, only the three best candidates can be chosen. Top-p (nucleus sampling) limits sampling to the smallest set of tokens whose probabilities add up to p; with top-p of 0.9, the model chooses only among tokens that together make up 90 out of every 100 parts of the probability. Top-p adapts to the situation: when the model is confident, few tokens qualify; when it is unsure, more do.

# Which candidate tokens survive top-k and top-p filters?
probs = {"durable": 0.55, "rugged": 0.25, "lightweight": 0.12, "comfy": 0.05, "sparkly": 0.03}
ranked = sorted(probs.items(), key=lambda kv: kv[1], reverse=True)

def top_k(k):
    return [w for w, _ in ranked[:k]]

def top_p(p):
    kept, running = [], 0.0
    for w, pr in ranked:
        kept.append(w); running += pr
        if running >= p: break
    return kept

print("top-k 2:", top_k(2))
print("top-p 0.8:", top_p(0.8))
print("top-p 0.95:", top_p(0.95))

top-k 2: ['durable', 'rugged']
top-p 0.8: ['durable', 'rugged']
top-p 0.95: ['durable', 'rugged', 'lightweight', 'comfy']

Maximum output tokens caps how long a response can be, which controls cost and stops runaway answers; set it too low and answers are cut off mid-sentence. Stop sequences are strings that end generation when produced, useful when an application expects a fixed format. Safety settings adjust how strictly the service blocks content in categories such as harassment, hate speech, sexually explicit material and dangerous content; Birchline keeps them at the defaults or stricter for anything customer-facing.

Ravi's rule of thumb is to change one setting at a time and evaluate the effect on real examples. Providers tune defaults sensibly, and many problems blamed on settings are really problems with the prompt or the data.

Determinism and repeatability

Even at temperature zero, outputs can vary slightly between runs because of how computations are spread across hardware, and models are periodically updated. Leaders should not design processes that assume a model will give exactly the same words every time. When repeatability matters, such as for classification labels or extracted fields, use low temperature, ask for structured output with a fixed set of allowed values, pin a specific model version where the platform allows it, and validate every response in code.