Retrieval-augmented generation

Jonah asked the model in the sandbox a simple question: "What is Cascade Ridge's late cancellation fee for surgery appointments?" The answer came back immediately and confidently. It named a fee, a notice period and an exception for emergencies. Every detail sounded plausible. None of it matched Cascade Ridge's actual policy, which the model had never seen.

"That is the most important demonstration in this course," Noor said. "The model is not lying. It is doing what it was trained to do: produce likely-sounding text. It knows what cancellation policies usually say. It does not know what ours says. If we want answers from our documents, we have to give it our documents."

Retrieval-augmented generation, usually shortened to RAG, does exactly that. Before the model answers, the application searches a collection of trusted documents for the passages most relevant to the question and adds them to the prompt. The model then answers from those passages. This lesson explains how RAG works, what it is good for, and its limits. Lesson 6 covers the storage and the managed service, Amazon Bedrock Knowledge Bases, that Cascade Ridge used to build it.

Why foundation models need retrieval

A foundation model's knowledge has three gaps that matter to a business.

• It is frozen in time. The model learned from data up to a cutoff date. A policy updated last month is unknown to it.
• It has never seen private information. Cascade Ridge's SOPs, price lists and procedures were never in any public training data.
• It cannot show its sources. Even when a model states something true, it cannot say where it learned it, so staff cannot check.

Retrieval fills all three gaps. The documents can be updated at any time without retraining anything; they can be private; and the application knows which passages it supplied, so it can cite them. RAG also reduces hallucination, because the model has real text to draw on, though it does not remove it entirely.

How RAG works

A RAG system has two pipelines. The first runs ahead of time, whenever documents change. The second runs every time someone asks a question.

Ingestion prepares the documents:

• Load the source documents, such as the 140 SOP and policy files stored in an Amazon S3 bucket.
• Chunk them: split each document into passages of a manageable size, such as a few hundred tokens, often with a little overlap so a sentence split across two chunks is not lost.
• Embed each chunk: an embeddings model turns the text into a vector, a list of numbers that captures its meaning. AWS-112 introduced embeddings; lesson 6 shows how they are compared.
• Store the vectors, with the chunk text and metadata such as the document name, section and date, in a vector store.

Retrieval and generation answer a question:

• Embed the question with the same embeddings model.
• Search the vector store for the chunks whose vectors are closest to the question's vector. These are the passages most similar in meaning, even if they use different words.
• Augment the prompt: a template places the retrieved chunks and the question together, with instructions such as "Answer only from these passages."
• Generate: the foundation model writes the answer, and the application attaches citations to the passages used.

The word *augmented* is the key: the model's prompt is augmented with retrieved text. Nothing about the model itself changes.

Chunking in practice

Chunking sounds like a technical detail, but it has a large effect on answer quality. Chunks that are too large include irrelevant text and use up the context window; chunks that are too small lose the surrounding meaning, such as which policy a sentence belongs to. Jonah experimented with a short policy document to see how chunk size and overlap change the pieces.

policy = ("Late cancellation. Clients who cancel a routine appointment with less than 24 hours notice "
          "may be charged a fee. Surgery appointments need 48 hours notice. The fee is waived for "
          "emergencies and for a first late cancellation in any twelve months. Staff record the reason "
          "in the client file.")
words = policy.split()

def chunk(words, size, overlap):
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

for size, overlap in [(20, 0), (20, 5)]:
    parts = chunk(words, size, overlap)
    print(f"size {size}, overlap {overlap}: {len(parts)} chunks")
    for p in parts:
        print("  -", p)

size 20, overlap 0: 3 chunks
  - Late cancellation. Clients who cancel a routine appointment with less than 24 hours notice may be charged a fee. Surgery
  - appointments need 48 hours notice. The fee is waived for emergencies and for a first late cancellation in any twelve
  - months. Staff record the reason in the client file.
size 20, overlap 5: 3 chunks
  - Late cancellation. Clients who cancel a routine appointment with less than 24 hours notice may be charged a fee. Surgery
  - be charged a fee. Surgery appointments need 48 hours notice. The fee is waived for emergencies and for a first
  - emergencies and for a first late cancellation in any twelve months. Staff record the reason in the client file.

With no overlap, the second chunk begins in the middle of the surgery sentence, so a search for "surgery notice" might retrieve a chunk that no longer says what the notice period applies to. A small overlap keeps the sentence together in at least one chunk. Real systems also chunk by structure, splitting at headings or paragraphs, so each chunk is a meaningful unit. Amazon Bedrock Knowledge Bases offers several chunking strategies, including fixed-size chunks, hierarchical chunking that keeps small chunks linked to their larger parent sections, and semantic chunking that splits where the topic changes. For Cascade Ridge's SOPs, which were already written with clear headings, chunking at section boundaries worked best.

Business applications of RAG

RAG is the most common way businesses put foundation models to work on their own knowledge. Typical uses include:

• Internal question answering over policies, procedures and manuals, like Front Desk Helper.
• Customer support that answers from product documentation and help articles, with links to the source.
• Research assistants that find and summarize relevant passages from large collections of reports or contracts.
• Grounded drafting, such as writing a client letter that quotes the correct clinic hours and policy wording.

In each case the value comes from three properties: answers reflect current, private information; they can be checked against cited sources; and the content can be updated by updating documents, not models.

The limits of RAG

RAG is powerful, but it has failure points, and a practitioner should know where they are.

• Retrieval misses. If the right passage is not retrieved, the model cannot use it. A question phrased very differently from the document, or a poorly chunked document, can cause misses.
• Bad sources give bad answers. If an SOP is out of date or two documents contradict each other, RAG will faithfully reflect the problem. Cascade Ridge found three pairs of conflicting SOPs during ingestion and had the operations team resolve them.
• The model can still go beyond the passages. Instructions such as "answer only from the passages" and a contextual grounding check in Amazon Bedrock Guardrails help, but evaluation is still needed.
• Permissions must be respected. If some documents are for managers only, the retrieval step must not hand them to everyone. Knowledge bases support metadata filters, so a query can be limited to documents a user is allowed to see.

"RAG moves the hard part," Noor told Jonah. "Instead of hoping the model knows the answer, we have to make sure the right passage is found and the model sticks to it. Both of those are things we can measure."

RAG compared with a long context window

Some foundation models accept very long prompts, enough to hold dozens of documents at once. Jonah asked whether that made retrieval unnecessary: why not paste every SOP into each request? Noor ran the numbers with him. The 140 documents came to roughly 210,000 tokens. Sending them with each of 400 daily questions would multiply input tokens, and therefore cost, by about a hundred compared with sending four retrieved passages, and every answer would take longer to start. Answers could also suffer, because models are better at using a few relevant passages than at finding one sentence in a very long prompt.

Long context windows are still useful: for summarizing a single long report, comparing two contracts, or answering questions about one document a user has just uploaded. The design question is the same as everywhere else in this course. Which approach gives good enough answers at acceptable cost and speed? For a large, changing document collection queried many times a day, retrieval almost always wins.

Grounding instructions

Retrieval only helps if the model uses what it is given. Three instructions in the prompt make a large difference: answer only from the provided passages; say clearly when the passages do not contain the answer; and cite the passage used. Front Desk Helper's template also told the model what to do when two passages disagreed: report both and suggest checking with the clinic manager. During testing this caught two of the three conflicting SOP pairs before the operations team had resolved them.

RAG also changed how the operations team thought about its documents. Because staff would now see the exact passage behind every answer, vague or outdated wording became visible in a way it never had been. Andre's team rewrote eleven SOPs during the prototype, mostly by adding clear headings, removing duplicated sections and putting the effective date at the top of each document. Better documents made better answers, and they also helped the staff who read the SOPs directly.