Human evaluation and benchmark datasets

The automatic scores for Night Notes looked excellent: high ROUGE, high faithfulness from the judge model, and correct format on 148 of 150 test notes. Noor still would not let it near a pilot. "Numbers tell us the summaries look like the references," she said. "They do not tell us whether a vet at three in the morning would trust one, or miss something important because of one. For that, we need vets."

Dr. Simone Kaur agreed to recruit four emergency veterinarians and two registered technicians for a review. This lesson explains human evaluation, when it is essential, how to design it so the results mean something, how benchmark datasets fit in, and how to decide whether a foundation model application actually meets its business objectives.

Human-in-the-loop evaluation

Human evaluation asks people to judge model outputs. Human-in-the-loop is a broader idea: people take part in the system at chosen points, whether to evaluate outputs during development, to review a sample in production, or to approve each output before it is used. The exam guide lists human-in-the-loop evaluation alongside benchmark datasets and Amazon Bedrock Model Evaluation as ways to evaluate foundation model performance.

Human evaluation is slower and more expensive than automatic metrics, so teams use it where it adds the most value:

• When the stakes are high. Clinical summaries, legal text, financial advice and anything that could harm someone need expert review.
• When quality is subjective. Tone, helpfulness, clarity and whether an answer would satisfy a real user are hard to score automatically.
• To calibrate automatic measures. Comparing human ratings with an LLM judge's scores on the same sample shows whether the judge can be trusted for the rest.
• To discover new failure types. People notice problems that no metric was designed to catch.

Designing a human evaluation

A poorly designed review produces opinions, not evidence. Noor and Dr. Kaur designed the Night Notes review with five rules.

• Clear criteria with a rating scale. Each reviewer rated each summary on four questions, using a scale from 1 to 5 with written descriptions for each score: Is every fact in the summary supported by the note? Is any important fact missing? Would the summary help you decide what to ask the owner? Is anything in it potentially misleading?
• Representative samples. Reviewers saw 60 summaries drawn from the test set, including the hard cases, in random order.
• Blind comparison where possible. For part of the review, each vet saw two summaries of the same note, one from the current version and one from a candidate, without knowing which was which, and chose the better one.
• More than one reviewer per item. Each summary was rated by at least two people, so disagreements could be seen and discussed.
• A written record. Every rating and comment was saved, so the results could be compared with later versions.

Agreement between reviewers is itself a useful measure. If two experts often disagree about whether a summary is good, the criteria may be unclear, or the task may be genuinely ambiguous. Jonah calculated simple agreement on the "important fact missing" question.

# Each pair is (reviewer A says a fact is missing?, reviewer B says a fact is missing?) for 60 summaries.
pairs = [(False, False)] * 47 + [(True, True)] * 7 + [(True, False)] * 4 + [(False, True)] * 2
agree = sum(a == b for a, b in pairs)
print("Summaries rated:", len(pairs))
print("Reviewers agreed on:", agree, "->", round(agree / len(pairs), 2))
flagged_by_anyone = sum(a or b for a, b in pairs)
print("Summaries flagged by at least one reviewer:", flagged_by_anyone)

Summaries rated: 60
Reviewers agreed on: 54 -> 0.9
Summaries flagged by at least one reviewer: 13

The reviewers agreed on 54 of 60. Thirteen summaries were flagged by at least one reviewer as missing an important fact, far more than the automatic scores suggested. Most involved timing: the note said when a toxin was eaten relative to the call, and the summary gave a time without saying "before the call". Vets care intensely about how long ago something happened. The fix, a rule in the template requiring times to be written relative to the call, came straight from the review, and no automatic metric had pointed to it.

Tools for human review on AWS

Amazon Bedrock Model Evaluation supports human evaluation jobs: you provide the prompts and model outputs or let the job generate them, define the rating criteria, and assign a work team of your own employees or subject-matter experts, who rate responses in a web interface. The results appear in the same kind of report as automatic jobs, so human and automatic scores can be compared side by side.

Beyond evaluation, AWS offers services for building human review into a system. Amazon SageMaker Ground Truth manages human labelling and review work, including labelling data for training and collecting human preference data. Amazon Augmented AI (Amazon A2I) routes selected model predictions to human reviewers, for example when a model's confidence is low. These matter for the exam mainly as examples of human-in-the-loop design: some outputs should be checked by a person before they are used.

Benchmark datasets

A benchmark dataset is a standard, published set of tasks with known answers, used to compare models on the same footing. Public benchmarks test general abilities such as reading comprehension, reasoning, coding, mathematics, multilingual understanding, and the tendency to produce toxic or biased text. Model providers report benchmark scores, and Bedrock Model Evaluation offers built-in datasets for common tasks.

Benchmarks are useful for narrowing a long list of candidate models, but they have well-known limits:

• They may not match your task. A model that scores well on general reasoning may still summarize veterinary call notes poorly.
• They can be contaminated. If a benchmark's questions appeared in a model's training data, its score may overstate its real ability.
• They can be optimized for. When a benchmark becomes a target, providers can tune for it, and scores rise faster than real-world performance.
• They age. As models improve, older benchmarks stop separating good models from great ones.

The practical answer is to use public benchmarks for a first shortlist and then build a custom benchmark from your own task, which is exactly what the 150-note and 200-question test sets are. A custom benchmark should be representative, include hard and edge cases, be kept private so it is not accidentally trained on, and grow over time as new failures are found.

Does it meet the business objective?

Good scores on a test set do not prove that an application is worth having. The exam guide asks whether a foundation model effectively meets business objectives, giving examples such as productivity, user engagement and task engineering, and it lists business alignment metrics such as task completion rate, user satisfaction and cost per interaction.

• Productivity: does the application save time? For Night Notes, how many seconds does a vet save reading a summary instead of the raw note, without missing anything?
• User engagement: do people actually use it, and keep using it? A tool staff abandon after a week has failed, whatever its scores.
• Task completion rate: how often does the user achieve what they came for? For Front Desk Helper, how often does a staff member get a usable answer without escalating to a manager?
• User satisfaction: ratings, comments and short surveys.
• Cost per interaction: total cost divided by the number of interactions, compared with the value each provides. Lesson 11 shows how to track it.

Task engineering means designing the task itself so the model's strengths are used and its weaknesses are covered: splitting a large task into steps, deciding which parts a person does, and choosing where the model's output feeds into existing work. Night Notes is a good example. The model summarizes; the vet decides. The original note is always one tap away. That design matters as much as any score.

Noor wrote the pilot success measures with Dr. Kaur and Andre before the pilot began, so nobody could move the goalposts afterwards.

Bias and fairness in human review

Human reviewers bring their own biases. They may rate longer answers more highly, favour answers that match how they would have written them, or grow tired and less careful after many items. Good designs reduce these effects: short sessions, items in random order, blind comparisons, clear written criteria and a few known test items mixed in to check that reviewers are paying attention. Reviewers should also reflect the people who will use the output. For Front Desk Helper, that meant receptionists as well as managers, because a manager might understand an answer that a new receptionist would find confusing.

Human review also needs to be safe for the reviewers. Tasks that involve distressing material, such as reviewing content filters, should be limited, supported and shared. Cascade Ridge's reviews did not involve such material, but Noor mentioned it because larger evaluation programmes must plan for it.

The results of human review should flow back into the design. Each reviewer comment was tagged with a cause, such as missing fact, wrong emphasis or unclear wording, and the most common causes were fixed first. The next review then checked whether those causes had become less common, closing the loop.