Embeddings and semantic similarity

The sixth AI hour falls on December 7, and Maya brings a problem from the contact centre's knowledge base. Agents search it dozens of times a day, and they hate it. A search for "money back" finds nothing, because the policy page says "refund". A search for "broken on arrival" misses the page titled "Damaged items". Agents have learned to try three or four different words for everything. "The search only matches words," Maya says. "It doesn't understand what we mean." Arjun tells her that the technology that fixes this is also the foundation of Ask Finch: embeddings.

Meaning as numbers

An embedding is a list of numbers, called a vector, that represents the meaning of a piece of text, an image or another item. An embedding model is trained so that items with similar meanings get vectors that are close together, and items with different meanings get vectors that are far apart. "I want my money back" and "How do I get a refund?" share almost no words, but their embeddings are close, because they mean nearly the same thing. "How do I get a refund?" and "How do I get to the store?" share several words but point to different meanings, so their embeddings are further apart.

Real embeddings have hundreds or thousands of dimensions; a common size is 1,536 numbers per vector. No person can picture that, but the idea works the same way in two dimensions, where each text becomes a point on a map and similar meanings cluster together.

Embeddings are produced by embedding models, which are separate from chat models, cheaper to run and available in the Foundry model catalog. You send text to an embedding model and receive its vector. Inside a chat model, a similar process happens to every token as the first step of processing, which is why lesson 4 mentioned that tokens are turned into lists of numbers.

Measuring similarity

To compare two embeddings, systems usually use cosine similarity, which measures the angle between two vectors. If two vectors point in exactly the same direction, the cosine similarity is 1. If they are at right angles, unrelated, it is 0. If they point in opposite directions it is minus 1, although with typical text embeddings most scores fall between 0 and 1. The calculation multiplies the vectors element by element, adds up the products, and divides by the lengths of the two vectors.

import math
def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

# Tiny made-up 4-number embeddings; real ones have hundreds or thousands of numbers.
vec = {
    "How do I get a refund?":          [0.90, 0.10, 0.05, 0.20],
    "I want my money back":            [0.85, 0.15, 0.10, 0.25],
    "How do I get to the store?":      [0.10, 0.90, 0.20, 0.05],
    "My drill arrived broken":         [0.40, 0.05, 0.85, 0.10],
}
q = "How do I get a refund?"
for text, v in vec.items():
    if text != q:
        print(f"{cosine(vec[q], v):.3f}  {text}")

0.994  I want my money back
0.232  How do I get to the store?
0.487  My drill arrived broken

The refund question scores very close to "I want my money back", much lower against the broken drill, which is related because damaged items are often refunded, and lowest against directions to the store, despite sharing the words "How do I get". That is semantic similarity: closeness of meaning rather than overlap of words.

Semantic search

Semantic search, also called vector search, uses embeddings to find the documents whose meaning best matches a query. It works in two stages. Ahead of time, every document in the collection is split into passages, each passage is embedded, and the vectors are stored in an index designed for fast similarity search. At query time, the query is embedded with the same model, and the index returns the passages whose vectors are closest. In Azure, Azure AI Search provides vector indexes, and many databases, such as Azure Cosmos DB and Azure Database for PostgreSQL, can store and search vectors too.

Many systems combine semantic search with ordinary keyword search, an approach called hybrid search, because keywords are still better for exact matches such as product codes and model numbers. A search for "HF-20417-B" should find that product's page exactly, not pages about similar drills. Some systems add a final reranking step, in which a separate model reads the top results and reorders them by how well they answer the question.

Chunking

Before documents can be embedded they are split into chunks, passages of a manageable size such as a few hundred tokens. Chunk size is a trade-off. Chunks that are too large mix several topics, so their embeddings are vague and a search may return a long passage in which the answer is buried. Chunks that are too small lose context, so a passage that says "within this period" no longer says which period. Common practice is to split at natural boundaries such as headings and paragraphs, and to let neighbouring chunks overlap a little so that sentences near a boundary appear in both.

# Split a policy page into overlapping chunks of about 60 words (real systems count tokens).
page = ("Unused items in original packaging can be returned within the return period with a receipt. "
        "Opened items can be exchanged for the same product if they are not damaged. "
        "Special orders, cut lumber and mixed paint cannot be returned. "
        "Damaged items should be reported within seven days of delivery with a photo. "
        "Refunds go back to the original payment method within ten business days. ") * 3
words = page.split()
size, overlap = 60, 10
chunks = [words[i:i + size] for i in range(0, len(words), size - overlap)]
print("words on page:", len(words))
print("chunks:", len(chunks))
print("first chunk ends with:", " ".join(chunks[0][-6:]))
print("second chunk starts with:", " ".join(chunks[1][:6]))

words on page: 192
chunks: 4
first chunk ends with: back to the original payment method
second chunk starts with: a photo. Refunds go back to

Maya tries the idea on paper. She writes ten real searches her agents typed last week and, next to each, the page that actually answered it. In six of the ten, the words in the search appear nowhere on the right page: "money back" against "Refunds", "came smashed" against "Damaged items", "swap colour" against "Exchanges". Keyword search could only ever find those pages by luck. Semantic search is designed for exactly this gap between the words customers use and the words policies use, which is why Arjun calls it the most useful idea in the course for a contact centre.

Other uses of embeddings

Embeddings are useful far beyond search. They can group similar items: clustering a month of customer emails by embedding reveals the themes customers write about, without anyone defining the categories first. They can power recommendations, by suggesting products whose descriptions are close to what a customer just bought. They can detect duplicates, such as two supplier listings for the same item with different wording. They can classify, by comparing a new email with example emails for each topic and picking the closest. And multimodal embedding models place images and text in the same space, so that a photo of a garden hose can be matched with the text "garden hose".

How embedding models learn

Maya asks how a model learns that "refund" and "money back" belong together. Arjun explains that embedding models are usually trained on huge numbers of pairs of texts that belong together, such as a question and its answer, a title and its article, or two phrasings of the same idea, along with texts that do not belong together. Training pulls the vectors of matching pairs closer and pushes mismatched ones apart. After enough examples, the model places any new text near others with related meaning, even phrasings it never saw. The same approach, with pairs of images and captions, produces the multimodal embeddings that let a photo be matched with words.

Because closeness is learned from examples, it reflects how language is used in the training data rather than a dictionary. That usually works well, but domain words can surprise. To a general embedding model, "plumber's putty" and "silicone caulk" may look almost identical, while to a store associate they are used for different jobs. Testing with real queries catches such cases, and hybrid search with keywords softens them.

Size, cost and storage

Embeddings are cheap to create compared with chat responses, but a large collection adds up, and the vectors must be stored and searched. Arjun works out the size of the contact centre knowledge base and the product manuals together.

# Storage for an embedding index (illustrative sizes)
chunks = 2400 + 18600          # knowledge base chunks + product manual chunks
dimensions = 1536              # numbers per vector
bytes_per_number = 4           # 32-bit floating point
total_bytes = chunks * dimensions * bytes_per_number
print("chunks:", chunks)
print(f"vector storage: {total_bytes / 1_000_000:.1f} MB before index overhead")

chunks: 21000
vector storage: 129.0 MB before index overhead

About 129 megabytes is small by cloud standards, but Arjun notes that vector indexes keep extra structures to make search fast, and that some services charge by index size. Some embedding models allow shorter vectors, trading a little accuracy for less storage and faster search, a choice worth testing on large collections.

Practical points

A few rules keep embedding systems working. The query and the documents must be embedded with the same model; vectors from different models are not comparable. When the embedding model is changed, the whole collection has to be embedded again. The index must be kept up to date: when the returns policy changes, its old chunks must be removed and new ones added, or the search will keep returning the outdated text. And embeddings carry the patterns of their training data, including some biases, so search results should be tested with the kinds of queries real users type.

Arjun proposes a quick win before Ask Finch is even built: replace the contact centre's keyword search with hybrid search over the same knowledge base. Maya likes the idea because her agents will get better search within weeks, and the same index will later ground Ask Finch. She asks how they will know it is better. Arjun suggests taking fifty real searches agents made last month, noting which page was the right answer for each, and measuring how often the right page appears in the top three results with the old search and with the new.