Hallucination and how to reduce it

The sock sentence would not leave Dr. Kaur alone. "If the summary says the dog ate a sock," she explained to the sixth AI hour, "the overnight vet may plan for an obstruction and order imaging first. If it says the dog may have eaten a sock, the vet asks the owner more questions. One word changes the plan."

Noor agreed that this was the heart of the matter. Generative models produce fluent text that fits learned patterns, and sometimes the text that fits best is not what the source said. This lesson defines hallucination, explains why it happens, and works through the layered set of defenses that reduce it: grounding the model in trusted sources, clear prompts, conservative settings, automated checks such as the contextual grounding check in Amazon Bedrock Guardrails, and human review. No single defense is enough, and none brings the rate to zero, so the lesson ends with how to design a workflow that catches what slips through.

What hallucination is

A hallucination is output that is not supported by the model's input or by fact, presented as if it were. Researchers distinguish two broad kinds, and both appeared in the Cascade Ridge tests.

• Unfaithful to the source. The model contradicts or goes beyond the material it was given. Changing "may have eaten" to "ate", adding a weight the caller never mentioned, or naming the wrong pet are all unfaithful to the call note.
• Not factual. The model states something false about the world, such as a non-existent drug interaction, an invented study, or a fee schedule it never saw.

Hallucinations range from harmless to dangerous. A wrong day of the week in a newsletter draft is embarrassing. A confident but invented dosage instruction in a client message could harm an animal. The response to hallucination should therefore scale with the consequences of the task, which is why the overnight summaries got far more scrutiny than the newsletter.

Why models hallucinate

Hallucination is not a bug that a patch will remove. It follows from how generative models work.

• They generate, they do not retrieve. As lesson 3 showed, a language model picks a likely next token. If the most fluent continuation includes a detail that was not in the input, the model writes it.
• Training rewards plausible text. Models learn from text in which statements are usually confident. They learn to sound sure, not to measure their own certainty.
• Gaps in knowledge. When asked about something outside its training data or its input, such as Cascade Ridge's own fees, the model fills the gap with a plausible guess rather than saying it does not know.
• Ambiguous or overloaded prompts. Vague instructions, very long inputs, or questions that assume a false premise ("Why does our after-hours fee double on Sundays?") invite invention.
• Sampling. Higher temperatures increase the chance of picking less likely, and less accurate, tokens.

A layered defense

Because no single technique removes hallucination, practitioners stack several. Noor drew the layers on the board, and they became the design for the overnight summaries.

Layer 1: ground the model in trusted sources

Grounding means giving the model the facts it needs inside the request, so it summarizes or answers from those facts instead of from memory. For the overnight summaries, grounding is natural: the whole call note is the input. For a question about the after-hours fee, grounding means retrieving the current fee schedule and including it with the question. Lesson 2 described the machinery: chunk the documents, embed them, retrieve the relevant chunks. The approach is called retrieval augmented generation (RAG), and Amazon Bedrock Knowledge Bases provides a managed version.

Grounding is the single most effective defense for questions about an organization's own information, because it replaces guessing with reading. It is not a complete defense: a model can still misread or embellish the documents it was given, as the sock sentence proved.

Layer 2: write prompts that discourage invention

How the request is written changes how often the model invents. The instructions Noor and Jonah settled on for the summaries are typical of good practice.

• Use only the information in the call note. Do not add facts.
• Keep the caller's words of uncertainty, such as "may have", "possibly" and "not sure".
• List every medication or substance mentioned, even if it seems minor.
• If something is unclear in the note, write "unclear in note" rather than guessing.
• Use the fixed headings: Patient, Concern, Timing, Actions taken by owner, Advice given.

Asking for a fixed structure helps twice: it reminds the model what to include, and it makes omissions easy for the reviewer to spot. For questions answered from documents, the equivalent instructions are to answer only from the supplied passages, to say "I don't know" when the passages do not contain the answer, and to cite which passage each statement comes from. The next course in this program covers prompt engineering in detail.

Layer 3: use conservative inference settings

For factual tasks, set a low temperature, so the model favors its most likely tokens, and a sensible maximum length, so it is not encouraged to pad. These settings reduce variation and embellishment; they do not make an unfaithful model faithful. Creative tasks such as newsletter headlines can use higher settings because invention there is the point.

Layer 4: check the output automatically

Automated checks run on every response, before a person sees it. Some are simple rules written by the team; others are managed features.

• Rule-based checks. For the summaries, a script confirms that every drug name in the note appears in the summary, and that hedge words in the note are not dropped. These are cheap and catch specific, known failure modes.
• Contextual grounding checks. Amazon Bedrock Guardrails includes a contextual grounding check that scores whether a response is grounded in the supplied source and relevant to the question, and can block or flag responses that fall below a threshold you set.
• Automated Reasoning checks. Also part of Bedrock Guardrails, these validate a response against a set of logical rules you define from a policy document, such as eligibility rules for a service plan.
• Model-based evaluation. A second model can be asked to compare the summary with the source and list unsupported statements. This catches more varied errors but is itself fallible and adds cost.

# A rule-based check: flag key terms in the source that the summary drops,
# and words in the summary that never appear in the source.
source = ("Owner of Biscuit (beagle, 6 y) says he may have eaten a sock this evening. "
          "Owner gave an antacid at 9 pm. Vomited once. Advised to come in now.")
summary = "Biscuit, beagle: ate a sock this evening, vomited once. Owner coming in now."

def words(t):
    return {w.strip(".,:;()").lower() for w in t.split()}

must_keep = {"may", "antacid", "vomited", "sock"}
missing = sorted(w for w in must_keep if w in words(source) and w not in words(summary))
new = sorted(words(summary) - words(source) - {"", "biscuit"})
print("dropped key terms:", missing)
print("summary words not in source:", new)

dropped key terms: ['antacid', 'may']
summary words not in source: ['ate', 'coming']

The check flagged both problems from Dr. Kaur's review in one run: "may" and "antacid" were dropped. It also listed "ate", which is exactly the word that changed the meaning. Noor warned that such scripts are narrow, they catch what they are written to catch, but cheap, fast checks like this are worth having alongside managed features.

Layer 5: keep a person in the loop where errors matter

Human review, often called human-in-the-loop, means a qualified person checks AI output before it is acted on. For the overnight summaries, the morning vet reads each summary with a link to the full note, and the AI rule agreed with Dr. Kaur means the summary never sets a triage priority. Human review is the last line of defense and the most important one for consequential tasks.

Review only works if it is designed to work. Reviewers tire of checking output that is almost always right and start skimming, a pattern called automation bias. The team counter-measures were concrete: the full note is always one click away, summaries highlight medications and hedges in bold for quick checking, a sample of summaries is audited against full notes each week, and any miss found is logged and fed back into the tests.

Choosing the right tasks in the first place

The most effective way to avoid harmful hallucinations is to avoid using generative output where an invention would be dangerous and unreviewed. That is why Cascade Ridge's design principles, set in earlier sessions, matter here: exact figures such as doses and invoice amounts come straight from source systems; generated text never sets a clinical priority; and anything sent to a client either comes from approved text or is reviewed first. Lesson 12 applies the same principles to a client-facing assistant.

Prompting for honesty

Two prompt techniques deserve special mention because they target hallucination directly. The first is giving the model an explicit way out. Models are trained to be helpful, and without permission to say "I don't know" they tend to produce an answer anyway. An instruction such as "If the passages do not contain the answer, reply exactly: Not found in the manual" gives the model a clear, safe alternative, and it makes those cases easy to count. The second is asking for evidence: requiring each statement to name the passage it came from makes unsupported statements easier for both automated checks and reviewers to spot, and models instructed to cite tend to stay closer to their sources.

Jonah tested both on the procedure manual questions. With a fixed "not found" reply allowed, questions about topics the manual does not cover, such as the after-hours fee, stopped producing invented answers in the test set and produced the fixed reply instead. With citations required, reviewers could check an answer in seconds by opening the cited section. Neither change cost anything except a few extra tokens of instructions.