Tokens, context windows and model parameters

On November 30 Arjun opens the AI hour by handing Maya a printed bill. It is the first month's invoice line for the test Foundry resource, and it is tiny, but it is measured in a unit she has never seen on an invoice: tokens. "Everything about these models is counted in tokens," he says. "What they read, what they write, how much they can hold in mind at once, and what we pay. If you understand tokens, you understand half of the settings in Foundry." This lesson covers tokens, the context window, the settings that shape a model's answers and the ways a model can be deployed.

What a token is

A token is a piece of text that a model treats as a single unit. Tokens are often whole short words, such as "the" or "drill", but long or unusual words are split into several pieces, and punctuation and spaces count too. "Unbelievable" might become three tokens; a product code such as "HF-20417-B" might become six. The exact split depends on the model's tokenizer, the component that turns text into tokens, and different model families use different tokenizers. For English text, a common rule of thumb is that one token is about four characters, or about three quarters of a word, so 100 tokens is roughly 75 words. Other languages, code and text full of numbers usually take more tokens per word.

# Rough token estimates using the rule of thumb of about 4 characters per token for English text.
texts = {
    "short question": "Can I return an opened cordless drill?",
    "returns policy page": "x" * 5200,       # stands in for a 5,200-character policy page
    "product manual": "x" * 48000,          # stands in for a 48,000-character manual
}
for name, t in texts.items():
    print(f"{name:20s} {len(t):6d} characters  about {round(len(t) / 4):6d} tokens")

short question           38 characters  about     10 tokens
returns policy page    5200 characters  about   1300 tokens
product manual        48000 characters  about  12000 tokens

Estimates are fine for planning, but Arjun warns that the only accurate count comes from the model's own tokenizer, and Foundry reports the real token counts for every request.

Why tokens matter

Tokens matter in four ways. First, cost: most model deployments charge per token, with separate prices for input tokens, the prompt the model reads, and output tokens, the response it writes; output tokens usually cost more. Second, speed: models generate one token at a time, so a long answer takes longer than a short one. Third, limits: every model has a maximum number of tokens it can handle in one request. Fourth, quota: Azure limits how many tokens per minute a deployment can process, so a busy application needs enough quota for its peak traffic.

The context window

The context window is the maximum number of tokens a model can consider at once, counting both the input and the output. Everything the model knows about the current conversation has to fit inside it: the system message that sets its instructions, the earlier turns of the chat, any documents supplied for grounding, the customer's latest question, and the answer it is about to write. Context windows have grown dramatically, from a few thousand tokens in early models to hundreds of thousands, or more, in some current models, but they are still finite.

When a conversation grows longer than the window, something has to go. Applications usually drop or summarize the oldest turns. That is why a long chat can "forget" what was said at the start: the model is not remembering anything between requests; the application resends the conversation each time, and what no longer fits is not sent. A model has no memory of its own between calls. Ask Finch will keep only the last few turns plus a short summary, and will start fresh for each new customer.

Bigger is not automatically better. A request that fills a huge context window costs more, takes longer, and can make it harder for the model to find the relevant part. Arjun's approach for Ask Finch is to send only the few passages of policy that relate to the customer's question, a technique explained in lessons 6 and 7.

Settings that shape the answer

When an application sends a request to a chat model, it can include settings, sometimes called inference parameters, that change how the model chooses each token. These are the configuration parameters the AI-901 outline refers to. The most important ones are listed below; not every model supports all of them, and reasoning models ignore some.

Temperature deserves a closer look. The model produces a score for every possible next token, and these scores are turned into probabilities. Temperature divides the scores before that conversion: a low temperature exaggerates the differences, so the top choice dominates; a high temperature flattens them, so unlikely tokens get a real chance. At a temperature of zero the model nearly always picks the single most likely token, which makes its answers more repeatable, though not guaranteed identical.

import math
# Scores a model might give four candidate next words after "Unused items can be returned within"
scores = {"30": 4.0, "90": 2.5, "60": 2.0, "14": 1.5}
def probs(temp):
    exps = {w: math.exp(s / temp) for w, s in scores.items()}
    total = sum(exps.values())
    return {w: round(e / total, 3) for w, e in exps.items()}
for t in (0.2, 1.0, 2.0):
    print(f"temperature {t}:", probs(t))

temperature 0.2: {'30': 0.999, '90': 0.001, '60': 0.0, '14': 0.0}
temperature 1.0: {'30': 0.694, '90': 0.155, '60': 0.094, '14': 0.057}
temperature 2.0: {'30': 0.47, '90': 0.222, '60': 0.173, '14': 0.135}

For Ask Finch, Arjun sets the temperature low. Customers asking about returns should get the same answer today and tomorrow. Marcus's product-description tool, by contrast, can use a moderate temperature so that a merchandiser can ask for two or three different drafts and choose the best.

Deploying a model

A model in the Foundry catalog cannot be used until it is deployed: set up as an endpoint that applications can call. When you deploy, you give the deployment a name, choose the model and version, choose a deployment type and set its quota. Applications then send requests to the deployment's endpoint, using a key or, better, a Microsoft Entra identity.

Deployment types differ in where requests are processed and how capacity is paid for. Standard deployments are billed per token. Global deployments can route requests to Microsoft data centers anywhere in the world for the best availability; data zone deployments keep processing within a geographic zone such as the United States or the European Union; regional deployments process in the chosen region. Provisioned throughput deployments reserve a fixed amount of capacity for a monthly or hourly charge, suited to steady, heavy workloads. Some models from other providers are offered as serverless APIs billed by Microsoft, and open models can also be run on managed compute that the customer sizes. The exact options vary by model and region, so always check the current list.

For Harrow & Finch, the review group is comfortable with processing anywhere in the United States, so Arjun plans a data zone deployment in the US geography, which balances availability with keeping data in the country.

az cognitiveservices account deployment list --name hf-foundry-test --resource-group rg-ai-test --output table

Rate limits and quota

Each deployment has a quota expressed in tokens per minute, and a related limit on requests per minute. If an application sends more than its quota allows, the service rejects the extra requests with a "too many requests" error and the application must wait and retry. Arjun estimates the pilot's need from Maya's figures: the chat channel receives about 600 conversations on a busy day, peaking at about 90 an hour, each with about four requests of around 4,500 tokens.

# Peak token demand for the Ask Finch pilot
conversations_per_hour = 90
requests_per_conversation = 4
tokens_per_request = 4500
per_minute = conversations_per_hour * requests_per_conversation * tokens_per_request / 60
print(f"peak tokens per minute: {per_minute:,.0f}")
print(f"with a safety margin of 1.5x: {per_minute * 1.5:,.0f}")

peak tokens per minute: 27,000
with a safety margin of 1.5x: 40,500

The result, about 27,000 tokens per minute at peak and about 40,500 with a margin, tells Arjun what quota to request. He also plans for what happens if the limit is reached anyway: the chat window will show a polite "one moment" message and retry, and if the delay grows, it will offer to connect the customer to an agent.

Streaming and response time

Because models generate one token at a time, a long answer can take several seconds to finish. Most chat applications therefore use streaming: the service sends tokens back as they are produced, and the chat window shows the answer appearing word by word. The total time is the same, but the customer sees progress at once, which feels much faster. Two measures matter for a responsive assistant: the time to the first token, which depends mostly on the size of the input and the load on the service, and the rate of tokens per second after that, which depends mostly on the model's size. Keeping prompts lean helps the first; choosing a smaller model helps the second. Maya, who knows how quickly customers abandon a chat, asks for a target of under two seconds to the first words, and Arjun adds it to the test plan.