Grounding and retrieval-augmented generation

In a test, Jules asks a general-purpose model, "What is Birchline's return window for footwear?" It answers confidently: thirty days. Birchline's actual policy is sixty days for unworn footwear and a separate warranty process for defects. The model had never seen Birchline's policy; it produced the most typical answer for retailers in general. "That," says Ravi, "is the problem grounding solves." This lesson explains grounding, retrieval-augmented generation and the kinds of data a model can be grounded in.

What grounding means

Grounding is connecting a model's output to verifiable sources of information, so that its answers are based on specific, relevant and current facts rather than only on what it absorbed during training. A grounded answer to the return question would be based on Birchline's actual returns policy, ideally with a citation showing which section it came from.

Grounding addresses three limitations from earlier lessons at once. It supplies knowledge the model never had, such as a company's private documents. It supplies current information past the model's training cutoff, such as this week's prices or today's news. And it reduces hallucination, because the model is instructed to answer from supplied evidence and to say when the evidence does not cover a question. Grounding does not make hallucination impossible, but it makes answers far more reliable and, with citations, checkable.

Kinds of grounding data

The exam guide distinguishes three sources of grounding data, and Ravi maps each to Birchline.

• First-party data is the organization's own information: Birchline's product catalog, policies, store inventory, order history, manuals and help desk articles. It is the most valuable for most business uses, because no public model has it.
• Third-party data comes from outside organizations under agreement: a supplier's product specifications, a weather service's forecasts, a mapping provider's store distances, or licensed industry data.
• World data is broad public information, such as what is on the open web. Grounding with Google Search lets a model draw on current public information, useful for questions such as recent trail closures or general camping advice.

Each source has its own trust and permission questions. First-party data must be accurate and access-controlled. Third-party data must be used within its license. World data must be treated with the caution any web source deserves. Many applications combine sources: the shopping assistant uses Birchline's catalog and policies for product answers and, for general questions about hiking conditions, may use Google Search grounding with links shown to the shopper.

Retrieval-augmented generation

The most common way to ground a model in an organization's documents is retrieval-augmented generation (RAG). Instead of hoping the model knows the answer, the application first retrieves the passages most relevant to the question from a collection of documents, then augments the prompt by inserting those passages, and finally asks the model to generate an answer using them.

• Retrieve: search the document collection for the passages most relevant to the user's question.
• Augment: place those passages in the prompt, with instructions such as "Answer only from the sources below and cite them."
• Generate: the model writes an answer based on the supplied passages, with citations.

RAG has big practical advantages. Documents can be updated without retraining any model: change the returns policy, re-index it, and the next answer uses the new wording. Access control can apply to retrieval, so a store associate's question only retrieves documents that associate is allowed to see. And citations let users check answers against the source, which builds trust and catches errors.

How retrieval finds relevant passages

Before retrieval can happen, documents are prepared. They are split into chunks, passages of a manageable size such as a few paragraphs, because sending whole manuals wastes tokens and buries the relevant sentence. Each chunk is then indexed so it can be found quickly.

Traditional keyword search finds chunks that contain the words in the question. It works well for exact terms such as product codes, but fails when the question and the document use different words: a shopper asks about "sending back boots" while the policy says "footwear returns". Semantic search solves this with embeddings: numerical vectors, lists of hundreds of numbers, that represent the meaning of a piece of text. An embedding model turns each chunk, and each question, into a vector, and texts with similar meanings end up with vectors that point in similar directions. Retrieval then finds the chunks whose vectors are closest to the question's vector. Those vectors are stored in a vector database or vector index built for fast similarity search.

import math
# Toy 4-dimensional "embeddings" (real ones have hundreds of dimensions).
# Dimensions loosely represent: returns, footwear, tents, shipping.
chunks = {
    "Footwear may be returned within 60 days if unworn.":        [0.9, 0.8, 0.0, 0.1],
    "Tents carry a two-year warranty against seam failure.":     [0.2, 0.0, 0.9, 0.0],
    "Standard shipping takes three to five business days.":      [0.1, 0.0, 0.0, 0.9],
    "Worn or defective footwear goes through the warranty desk.": [0.6, 0.9, 0.0, 0.0],
}
question = [0.8, 0.7, 0.0, 0.2]   # "Can I send back boots I bought last month?"

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

ranked = sorted(chunks, key=lambda c: cosine(question, chunks[c]), reverse=True)
for c in ranked[:2]:
    print(round(cosine(question, chunks[c]), 3), c)

0.995 Footwear may be returned within 60 days if unworn.
0.949 Worn or defective footwear goes through the warranty desk.

The question never uses the word "footwear", yet the two footwear passages score highest because their vectors point in a similar direction. Real systems use embedding models that capture meaning far more richly, and often combine semantic and keyword search (hybrid search) and then re-rank the top results with a model that judges relevance more carefully.

Writing the augmented prompt

Retrieval is only half of RAG. The augmented prompt must tell the model how to use the passages. Birchline's policy assistant uses instructions along these lines: answer only from the numbered sources; cite the source number after each claim; if the sources do not contain the answer, say "I could not find that in Birchline's policies" and suggest contacting the help desk; never guess. Those instructions turn a fluent general model into a careful specialist.

retrieved = [
    ("Returns policy 3.2", "Unworn footwear may be returned within 60 days with proof of purchase."),
    ("Warranty guide 1.4", "Worn or defective footwear is assessed by the warranty desk."),
]
question = "Can I return hiking boots I wore twice?"

prompt = ["Answer the question using only the sources below. Cite the source name in brackets.",
          "If the sources do not answer it, say you could not find it in Birchline's policies.", ""]
for i, (name, text) in enumerate(retrieved, start=1):
    prompt.append(f"Source {i} ({name}): {text}")
prompt += ["", "Question: " + question]
print("\n".join(prompt))

Answer the question using only the sources below. Cite the source name in brackets.
If the sources do not answer it, say you could not find it in Birchline's policies.

Source 1 (Returns policy 3.2): Unworn footwear may be returned within 60 days with proof of purchase.
Source 2 (Warranty guide 1.4): Worn or defective footwear is assessed by the warranty desk.

Question: Can I return hiking boots I wore twice?

Grounding on Google Cloud

Google Cloud offers grounding at several levels of effort, which GCP-111 covers in detail. At the simplest, grounding with Google Search connects Gemini models to current public web results, returning answers with links to sources. For an organization's own content, Agent Search on the Gemini Enterprise Agent Platform (the enterprise search service, which earlier materials called Vertex AI Search) provides a prebuilt RAG system: point it at websites, document stores or databases, and it handles chunking, indexing, retrieval and grounded answers. For teams that want more control, RAG APIs and vector search services let developers build custom pipelines with their own chunking, embeddings and ranking.

The choice follows a familiar pattern from GCP-101: prefer the most managed option that meets the need. Birchline's internal policy assistant is a good fit for prebuilt enterprise search over its policy documents. The shopping assistant, which must combine product search, live inventory and order status, needs more custom design, with retrieval for documents and tool calls for live systems (lesson 8).

Where RAG goes wrong

RAG is powerful but not magic, and Ravi lists the failures he has seen. Poor retrieval is the most common: if the right passage is not retrieved, the model cannot use it, so retrieval quality must be measured on its own. Bad chunking splits a rule from its exception, so the model sees half a policy. Stale or conflicting documents produce stale or conflicting answers; three versions of the returns policy in the index is a data governance problem, not a model problem. Weak instructions let the model blend retrieved facts with its own guesses. And missing permissions can leak documents to people who should not see them.

Each failure has a remedy: evaluate retrieval separately from generation, chunk along the document's structure, keep one authoritative version of each document, instruct the model to answer only from sources and to say when it cannot, and enforce access control at retrieval time. Jules adds a line to his notebook: "Grounding quality is mostly data quality."

Measuring retrieval on its own

Because poor retrieval is the most common RAG failure, Ravi measures it separately from the final answers. For each of fifty test questions, the team records which document passage actually contains the answer. A retrieval run then checks whether that passage appears among the top three or five results. If the right passage is missing for a question, no amount of prompt work will fix the answer; the fix lies in chunking, document quality or the search settings. If the right passage is retrieved but the answer is still wrong, the problem is in the prompt or the model. Separating the two saves days of guessing.

Retrieval quality also depends on the question. Shoppers write short, informal questions full of typos, while policies use formal language. Some systems therefore rewrite the user's question into a clearer search query before retrieval, or generate several alternative queries and combine the results.