Tokens, embeddings and vectors

Jonah arrived at the second AI hour with a printed invoice from a generative AI trial that Kwame had run on his own account. The charges were listed in a unit Jonah had never seen: thousands of input tokens and output tokens. "We pay for words?" he asked. "Not quite," Noor said. "We pay for tokens. And to understand tokens, we need to see how a model turns language into numbers."

Computers cannot do arithmetic on the word "vomiting". Every generative model therefore starts by converting text, images or sound into numbers, does all its work on those numbers, and converts the result back at the end. This lesson covers three ideas that make that possible and that appear constantly in exam questions and on invoices: tokens, embeddings and vectors. It also introduces chunking, the way long documents are cut into pieces a model can handle.

Tokens: the units a model reads and writes

A token is the basic unit of text a language model processes. Before a model sees a sentence, a component called a tokenizer splits it into tokens and replaces each one with a number, its token ID, from the model's vocabulary. The model reads a sequence of token IDs and, when it writes, produces token IDs that the tokenizer turns back into text.

Tokens are not the same as words. Common words are often a single token. Longer or rarer words are split into several pieces: "booking" might become "book" and "ing". Spaces and punctuation are usually part of tokens too. Names, technical terms and words in languages that were rare in the training data tend to break into more tokens. A drug name such as maropitant, which appears in veterinary notes but rarely on the general internet, may cost four or five tokens on its own.

As a rough guide for English text, one token averages about three quarters of a word, so 100 tokens is roughly 75 words. Noor stressed the word rough. Every model family has its own tokenizer, so the same paragraph produces a different token count on different models, and text in Spanish or with many numbers usually produces more tokens per word than plain English.

Tokens matter for three practical reasons, all of which the exam touches.

• Cost. Most generative AI services charge per token, with separate prices for input tokens (the prompt you send, including any documents) and output tokens (what the model writes). Output tokens usually cost more. Lesson 10 works through the arithmetic.
• Limits. Every model has a context window, the maximum number of tokens it can consider at once, counting both the input and the output. A long transcript plus instructions plus the answer must fit inside it.
• Speed. Models generate output one token at a time, so a longer answer takes longer to arrive. Asking for a three-sentence summary rather than a page makes the response faster and cheaper.

# A very rough token estimate: about 4 characters of English per token.
# Real counts come from the model's own tokenizer and will differ.
note = ("Caller: owner of Biscuit, 6-year-old beagle. Ate about half a box of "
        "raisins around 9 pm. No vomiting yet. Advised to come in now; on the way.")
chars = len(note)
words = len(note.split())
est_tokens = round(chars / 4)
print("characters:", chars)
print("words:", words)
print("rough token estimate:", est_tokens)
print("tokens per word (rough):", round(est_tokens / words, 2))

characters: 142
words: 27
rough token estimate: 36
tokens per word (rough): 1.33

Chunking: fitting long documents into a model

The Cascade Ridge procedure manual runs to more than three hundred pages. Even a model with a large context window should not be sent the whole manual with every question: it would be slow, expensive and, in practice, models pay less reliable attention to material buried in the middle of a very long input. The usual solution is chunking: splitting a long document into smaller pieces, called chunks, that can be stored and retrieved separately.

There are several ways to chunk. Fixed-size chunking cuts every few hundred tokens. Structure-aware chunking follows headings, paragraphs or sections, so each chunk is a complete thought, such as one procedure. Many systems add a small overlap, repeating the last sentence or two of one chunk at the start of the next, so that an idea split across a boundary is not lost. Chunk size is a trade-off: chunks that are too small lose context, and chunks that are too large dilute the relevant part with unrelated text.

Noor suggested chunking the manual by procedure, because each procedure, such as "after-hours controlled drug count", was already written to stand alone. A question about the drug count would then retrieve that one section, not three unrelated pages.

# Split a long text into chunks of at most N words, with a small overlap.
def chunk(text, size=12, overlap=3):
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        chunks.append(" ".join(words[start:start + size]))
        if start + size >= len(words):
            break
        start += size - overlap
    return chunks

manual = ("Count controlled drugs at the end of every overnight shift. Two staff sign the "
          "register. Any difference is reported to the medical director before handover. "
          "Store the register in the locked pharmacy cabinet.")
for i, c in enumerate(chunk(manual), 1):
    print(i, c)

1 Count controlled drugs at the end of every overnight shift. Two staff
2 shift. Two staff sign the register. Any difference is reported to the
3 reported to the medical director before handover. Store the register in the
4 register in the locked pharmacy cabinet.

Embeddings and vectors: meaning as numbers

Tokens let a model read text, but they say nothing about meaning on their own: token 8414 is just a number. Inside the model, and in many applications built around it, text is represented by embeddings. An embedding is a list of numbers, typically hundreds or a few thousand of them, that captures the meaning of a piece of text, an image or other content. Such a list of numbers is called a vector, and the number of values in it is its dimension.

The useful property of embeddings is that similar meanings produce similar vectors. "Reschedule my appointment" and "move Max's visit to Friday" share almost no words, but an embedding model places them close together, because they mean nearly the same thing. "Why is my bill so high?" lands somewhere else entirely. You can picture each embedding as a point in a space with many dimensions; the figure flattens that space to two dimensions to show the idea.

Embeddings are produced by embedding models, a kind of foundation model that outputs vectors instead of text. Amazon Bedrock offers embedding models from Amazon and other providers. You send in text, and you get back a vector that you can store and compare.

Measuring similarity

To use embeddings, you need a way to tell how close two vectors are. The most common measure is cosine similarity, which compares the directions of two vectors. A value near 1 means the meanings are very similar; a value near 0 means they are unrelated. Some systems use distance measures such as Euclidean distance instead, where smaller is closer. The exam does not ask you to calculate these by hand, but seeing the arithmetic once makes the idea concrete.

import math
def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

# Tiny made-up 4-dimensional embeddings (real ones have hundreds of values).
vec = {
    "reschedule my appointment": [0.82, 0.10, 0.05, 0.31],
    "move Max's visit to Friday": [0.78, 0.15, 0.02, 0.36],
    "why is my bill so high":     [0.08, 0.91, 0.12, 0.05],
    "how do I give the ear drops": [0.11, 0.07, 0.88, 0.20],
}
query = "reschedule my appointment"
for text, v in vec.items():
    if text != query:
        print(f"{cosine(vec[query], v):.3f}  {text}")

0.995  move Max's visit to Friday
0.218  why is my bill so high
0.252  how do I give the ear drops

Vector databases and semantic search

Once every chunk of the procedure manual has an embedding, the chunks and their vectors can be stored in a vector database or a vector index, a store designed to find the vectors closest to a query vector quickly, even among millions of entries. A question from a clinic manager is turned into an embedding with the same model, the database returns the nearest chunks, and those chunks can be shown to the manager or passed to a language model to write an answer.

This is called semantic search: search by meaning instead of by matching keywords. A manager who types "what do we do if the drug count is off" finds the controlled drug procedure even though it uses the words "register" and "difference". Passing the retrieved chunks to a language model to compose an answer is known as retrieval augmented generation, or RAG; the next course in this program covers it in depth.

On AWS, several services can store and search vectors. Noor listed the ones the exam is most likely to mention, and Jonah added them to his notes.

Putting the pieces together

By the end of the hour Jonah could trace what happens when a clinic manager asks the planned manual assistant a question. The manual has already been chunked by procedure. Each chunk has been turned into an embedding and stored in a vector index. The manager's question is tokenized, embedded and compared with the stored vectors; the closest chunks are retrieved. The question and the chunks are then sent to a language model as input tokens, and the model writes an answer as output tokens. Every step uses one of the ideas from this lesson.

Lorraine Beaudry, the CFO, stopped by long enough to hear the word tokens and asked for one number: how much would a question cost? Noor promised a worked answer in lesson 10, with one caution already: the documents sent along with the question are input tokens too, so good chunking is also good cost control.