Evaluating foundation model output

Jonah had a habit Noor wanted to break. After each change to Front Desk Helper, he typed three or four questions into the sandbox, read the answers, and declared the new version "better". At the ninth AI hour, Noor asked him how much better, on which kinds of questions, and whether anything had got worse. He did not know.

"Reading a few answers is a demo, not an evaluation," she said. "Generative models can be fluent and wrong in the same sentence. We need a fixed set of test cases, measures we can repeat, and a way to compare versions fairly. Then 'better' means something."

This lesson covers how to evaluate foundation model output: building a test set, the automatic metrics the exam names (ROUGE, BLEU, BERTScore and LLM-as-a-judge), Amazon Bedrock evaluations, and how to evaluate whole applications such as RAG systems and agents. Lesson 10 covers human evaluation and benchmark datasets in more depth.

Why evaluating generative output is hard

Evaluating a classifier, as in AWS-111, is relatively simple: each prediction is right or wrong, and accuracy, precision and recall follow. Generative output is different. A question can have many good answers worded differently, and an answer can be mostly right with one wrong detail. Evaluation therefore uses several kinds of measure, each catching different problems:

• Reference-based metrics compare the output with a human-written reference answer.
• Model-based judgement uses another model to score outputs against criteria.
• Human evaluation asks people, often experts, to rate or compare outputs.
• Task and business measures check whether the application achieves its purpose, such as staff finding the right answer faster.

All of them start with a test set, sometimes called a golden dataset: a fixed collection of realistic inputs, each with a reference answer or the facts a good answer must contain. Jonah and the clinic managers wrote 200 Front Desk Helper questions covering every SOP area, including 25 deliberately hard ones: questions with no answer in the documents, questions that should be refused because they are clinical, and questions whose answers had changed in the latest policy revision. Night Notes got 150 invented call notes with veterinarian-approved summaries, including the houseplant note from lesson 2.

Reference-based metrics

The exam guide names two classic metrics that compare words in the output with words in a reference.

• ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures how much of the reference appears in the output, by overlapping words or word sequences. It is recall-oriented: it asks whether the output captured what the reference said. It is most often used for summarization. ROUGE-1 counts single words, ROUGE-2 counts pairs of consecutive words, and ROUGE-L uses the longest common sequence.
• BLEU (Bilingual Evaluation Understudy) measures how much of the output appears in the reference, using overlapping word sequences of several lengths, with a penalty for outputs that are too short. It is precision-oriented and was designed for machine translation.

The code below computes simple versions of these ideas for one Night Notes summary. Real evaluations use standard libraries, but the counting is the same in spirit.

def words(t):
    return t.lower().replace(",", "").replace(".", "").split()

reference = "Dog ate one packet of sugar-free gum about 30 minutes ago and is drooling and unsteady."
output    = "Dog ate a packet of sugar-free gum 30 minutes ago, now drooling."

ref, out = words(reference), words(output)
overlap = sum(min(ref.count(w), out.count(w)) for w in set(out))

rouge1_recall = overlap / len(ref)       # share of the reference captured
unigram_precision = overlap / len(out)   # share of the output found in the reference
f1 = 2 * rouge1_recall * unigram_precision / (rouge1_recall + unigram_precision)

print("Overlapping words:", overlap)
print("ROUGE-1 recall:", round(rouge1_recall, 2))
print("Unigram precision (BLEU-style):", round(unigram_precision, 2))
print("F1:", round(f1, 2))
print("Exact match:", reference == output)

Overlapping words: 10
ROUGE-1 recall: 0.62
Unigram precision (BLEU-style): 0.83
F1: 0.71
Exact match: False

The summary scored well on precision, since almost everything it said was in the reference, but lower on recall: it dropped "unsteady", which a vet would want. That is useful. The weakness of word-overlap metrics is equally clear: "a packet" and "one packet" mean the same thing, but only exact words count. And a summary that said "Dog did NOT eat sugar-free gum" would share most of its words with the reference while meaning the opposite.

BERTScore

BERTScore addresses part of that weakness. Instead of matching exact words, it uses embeddings from a language model to compare the meaning of words in context, so "unsteady" and "wobbly" can count as a match. It correlates better with human judgement than pure word overlap for many tasks, but it still compares against a reference and still cannot judge facts it cannot see.

LLM-as-a-judge

LLM-as-a-judge uses a second foundation model to score each output against criteria written in plain language, such as correctness, completeness, faithfulness to the source, helpfulness and harmfulness. The judge can also explain its score. This scales much better than human review and can assess qualities that word overlap cannot, such as whether an answer stays within the provided passages.

Judges have weaknesses that must be managed. They can prefer longer answers, prefer outputs written in their own style, or be inconsistent. Good practice is to write clear scoring criteria with examples, check a sample of the judge's scores against human ratings, and avoid using the same model as both the system under test and its judge where possible.

Amazon Bedrock evaluations

Amazon Bedrock Model Evaluation runs these kinds of assessment as managed jobs. A team chooses the model, or a knowledge base, to test and the dataset, which can be its own prompt dataset in Amazon S3 or a built-in dataset, and the job produces a report. Evaluation types include:

• Automatic (programmatic) evaluation, which computes metrics such as accuracy, robustness and toxicity for common tasks like summarization, question answering and classification.
• Model as a judge, in which a judge model scores responses against chosen criteria and explains each score.
• Human evaluation, in which your own employees or a managed team rate responses through a work interface. Lesson 10 covers this.
• RAG evaluation for knowledge bases, which assesses both retrieval (did it find relevant passages?) and generation (is the answer correct, complete and faithful to them?).

Because evaluations can compare several models or prompt versions on the same dataset, they turn model selection from guesswork into a measured decision. Noor reran the Small, Medium and Large comparison from lesson 1 as an evaluation job on the 150-note test set. The Medium model still led, and the Small model's main failure was omitting secondary symptoms, a pattern the job's report made obvious.

Evaluating whole applications

An application can fail even when its model is good, so the exam guide also asks how to evaluate applications built with foundation models, such as RAG systems, agents and workflows.

• RAG applications: measure retrieval and generation separately. Context relevance asks whether the retrieved passages relate to the question; context coverage asks whether they contain what the answer needs; faithfulness (or groundedness) asks whether the answer sticks to those passages; and answer correctness compares with the reference. A wrong answer with irrelevant passages points to retrieval; a wrong answer with the right passages points to the prompt or model.
• Agents: check whether the task was completed, whether the right tools were called with valid arguments, how many steps it took, and whether any unsafe action was attempted.
• Workflows: test each step and the end-to-end result, because an error early in a chain is passed to every later step.

# Front Desk Helper test results (illustrative): one row per question.
results = [
    # (retrieved the right SOP section?, answer judged correct?)
    *[(True, True)] * 164, *[(True, False)] * 9, *[(False, False)] * 19, *[(False, True)] * 8,
]
n = len(results)
retrieval_hit = sum(r for r, _ in results) / n
correct = sum(a for _, a in results) / n
wrong_with_right_context = sum(1 for r, a in results if r and not a)
wrong_with_wrong_context = sum(1 for r, a in results if not r and not a)
print("Questions:", n)
print("Retrieval hit rate:", round(retrieval_hit, 3))
print("Answer correctness:", round(correct, 3))
print("Wrong despite right passages (prompt/model issue):", wrong_with_right_context)
print("Wrong with wrong passages (retrieval issue):", wrong_with_wrong_context)

Questions: 200
Retrieval hit rate: 0.865
Answer correctness: 0.86
Wrong despite right passages (prompt/model issue): 9
Wrong with wrong passages (retrieval issue): 19

Most of the wrong answers came with the wrong passages, so the next improvement belonged in retrieval, not the prompt. Jonah looked at the 19 failures and found that most were questions using clinic slang, such as "no-show fee", for a policy titled "missed appointment charge". Adding the common synonyms to each SOP's metadata and turning on hybrid search fixed most of them on the next run.

Building and maintaining a test set

A test set is only as good as its cases, and building one is real work. Noor gave Jonah four rules. Cover the real mix: draw questions from what staff actually ask, in proportion, not just the questions that are easy to write. Include hard and edge cases: questions with no answer, questions that must be refused, ambiguous questions and questions about recently changed policies. Write references carefully: for each question, record the correct answer and the SOP section it comes from, checked by someone who knows the policy. Keep it growing: every real failure becomes a new test case, so the same mistake is caught automatically next time.

The test set also needs protecting. If its questions and answers leaked into prompts as examples, or into any fine-tuning data, scores would rise without the application getting better. Cascade Ridge kept the test set in its own S3 bucket, readable only by the evaluation role, and the few-shot examples in prompts were written separately.

Finally, results need context. A correctness score of 0.86 means little on its own. Compared with the previous version, the cheaper model and the target agreed with the clinic managers, it becomes a decision.

Evaluation is also a conversation with the people who own the work. Jonah shared the failure list with the clinic managers each week. They often spotted patterns he had missed, such as a cluster of wrong answers that all came from one clinic's local variation of a policy.