Evaluating generative AI output

Two vendors visit Birchline in the same week, each promising that its approach to the shopping assistant is "the most accurate on the market". Jules asks both the same question: "Accurate at what, measured how, on whose data?" Neither has a good answer. Ravi is pleased. "That question is the whole of evaluation," he says. This lesson covers how to evaluate generative AI before launch and how to keep measuring it afterwards.

Why evaluation is different for generative AI

For a predictive model, evaluation is fairly straightforward: compare its predictions with known answers and count how often it is right. Generative output is harder. A product description can be correct in many different wordings; a summary can be accurate but miss the point; a chatbot answer can be true but unhelpful, or helpful but rude. There is rarely one right answer to compare against.

Generative AI evaluation therefore asks several questions at once. Is the output correct, meaning factually accurate? Is it grounded, supported by the supplied sources rather than invented? Is it relevant and complete for the request? Does it follow instructions on format, length and tone? Is it safe, free of harmful, biased or inappropriate content? And does it achieve the business goal, such as resolving the shopper's question without a handoff?

Building an evaluation set

Every evaluation starts with an evaluation dataset: a collection of realistic inputs, and where possible reference answers or criteria, that represents the work the system will do. Ravi and Jules build one for the policy assistant from three sources: 150 real questions from the help desk logs, with personal details removed; 30 questions written to test edge cases, such as items bought in another country or a policy that changed mid-season; and 20 questions the assistant should refuse or hand off, such as requests for legal advice.

A good evaluation set is representative of real traffic, includes hard and edge cases, is kept separate from any data used to tune the system, and is versioned, so results from different weeks are comparable. It grows over time: every real failure found in production becomes a new test case.

Ways to evaluate

There are three main ways to judge generative output, and serious projects use all three.

Human evaluation asks people to rate or compare outputs against criteria. It is the most trustworthy way to judge qualities such as helpfulness, tone and brand voice, but it is slow and costly. Rating guidelines must be clear, so that two reviewers would give the same score, and reviewers should not know which system produced which answer. Birchline's copy team rates description drafts on a five-point scale for accuracy and voice.

Automated metrics compute scores by program. For tasks with reference answers, metrics can measure overlap between the output and the reference; examples include BLEU, traditionally used for translation, and ROUGE, traditionally used for summaries. Classification tasks use standard measures such as accuracy, precision and recall. Simple automated checks are also valuable: is the output valid JSON, within the length limit, free of banned phrases, and does every cited source exist? Automated metrics are fast and cheap but can miss meaning; two summaries with little word overlap can both be excellent.

Model-based evaluation, often called LLM-as-a-judge or an autorater, uses a capable model to grade outputs against a rubric, for example "Rate from 1 to 5 how well this answer is supported by the sources, and explain why." Evaluations can be pointwise, scoring one output at a time, or pairwise, asking which of two outputs is better. Model judges scale well and handle nuance better than overlap metrics, but they have their own biases, such as favouring longer answers, so their scores should be checked against human ratings on a sample. Google Cloud's generative AI evaluation service, part of the Gemini Enterprise Agent Platform, provides model-based and computed metrics for comparing prompts, models and agents.

# Evaluate the email classifier against labelled test emails (sample results).
truth     = ["return", "order", "return", "product", "order", "return", "product", "order"]
predicted = ["return", "order", "product", "product", "order", "return", "product", "return"]

accuracy = sum(t == p for t, p in zip(truth, predicted)) / len(truth)
label = "return"
tp = sum(t == label and p == label for t, p in zip(truth, predicted))
fp = sum(t != label and p == label for t, p in zip(truth, predicted))
fn = sum(t == label and p != label for t, p in zip(truth, predicted))
precision = tp / (tp + fp)
recall = tp / (tp + fn)
print("accuracy:", accuracy)
print("for 'return': precision", round(precision, 2), "recall", round(recall, 2))

accuracy: 0.75
for 'return': precision 0.67 recall 0.67

For the returns queue, precision answers "of the emails sent to returns, how many belonged there?" and recall answers "of the real returns emails, how many reached the queue?" Which matters more is a business decision: Birchline cares most about recall for complaints, because a missed complaint is worse than an extra one in the queue.

Checking groundedness

For grounded systems, the most important single check is whether each claim in an answer is supported by the retrieved sources. Model-based judges are good at this, and simple programs can flag obvious problems, such as numbers in the answer that appear nowhere in the sources.

import re
sources = "Unworn footwear may be returned within 60 days. Worn footwear goes to the warranty desk."
answers = {"A": "You can return unworn boots within 60 days.",
           "B": "You can return boots within 90 days, worn or not."}

source_numbers = set(re.findall(r"[0-9]+", sources))
for name, text in answers.items():
    unsupported = [n for n in re.findall(r"[0-9]+", text) if n not in source_numbers]
    print(name, "unsupported numbers:", unsupported or "none")

A unsupported numbers: none
B unsupported numbers: ['90']

Answer B claims 90 days, a number the sources never mention, so it is flagged for review. Checks like this catch only some errors, which is why they are combined with model-based and human evaluation.

Comparing configurations

Evaluation is most useful as a comparison. A configuration is everything that shapes output: the model and version, the prompt, the generation settings, the retrieval setup and any tuning. Changing any of them can change quality. Ravi's team runs the whole evaluation set against each candidate and compares results side by side before choosing.

# Pairwise judge results on 40 policy questions: which answer did the judge prefer?
results = {"prompt v3 + large model": {"wins": 21, "ties": 7, "losses": 12},
           "prompt v3 + small model": {"wins": 12, "ties": 7, "losses": 21}}
for config, r in results.items():
    decided = r["wins"] + r["losses"]
    print(config, "win rate on decided pairs:", round(r["wins"] / decided, 2))
# Cost per answer (illustrative US dollars) puts the quality gap in context:
cost = {"prompt v3 + large model": 0.0045, "prompt v3 + small model": 0.0004}
print("cost ratio large to small:", round(cost["prompt v3 + large model"] / cost["prompt v3 + small model"], 1))

prompt v3 + large model win rate on decided pairs: 0.64
prompt v3 + small model win rate on decided pairs: 0.36
cost ratio large to small: 11.2

The large model wins more often, but it costs about eleven times as much per answer in this illustration. Whether the quality gain is worth it depends on the stakes: for customer-facing policy answers, Birchline accepts the cost; for internal email sorting, the small model's accuracy is already high enough.

Business measures and KPIs

Technical scores are means, not ends. Before launch, Jules defines key performance indicators (KPIs) that show whether the assistant helps the business: the share of conversations resolved without a handoff, customer satisfaction ratings, average time to answer, the rate of handoffs that should not have been needed, cost per conversation, and the number of reported errors. Each KPI gets a target and an owner. The exam expects leaders to connect generative AI projects to measurable outcomes like these, a topic GCP-112 develops.

Evaluating after launch: monitoring

Evaluation does not stop at launch. Real users ask questions no test set anticipated, documents change, and models are updated. Ongoing monitoring covers several things that the exam guide lists.

• Performance tracking: sample live conversations and score them on the same dimensions as the evaluation set, and watch the KPIs.
• Drift: watch for changes in the kinds of questions users ask, or gradual changes in output quality. A spike in questions about a new product line signals that the documents need updating.
• Versioning: record which model version, prompt version and data version produced each output, so problems can be traced and changes rolled back.
• Model upgrades: providers release new model versions and retire old ones on a schedule. Some services move to newer versions automatically. Rerun the evaluation set before adopting a new version, and pin versions where stability matters.
• Security patches and safety: keep screening and safety settings current, and watch for new misuse patterns such as fresh prompt injection attempts.
• User feedback: thumbs-up and thumbs-down ratings and written comments are cheap signals; route negative ones for review and add real failures to the evaluation set.

A/B testing in production

Offline evaluation predicts how a change will perform; production tells the truth. For customer-facing changes, Birchline runs A/B tests: a small share of conversations uses the new configuration while the rest use the current one, and the KPIs are compared after enough conversations to be confident in the difference. A new prompt that wins on the evaluation set but lowers the resolution rate in the A/B test is not adopted. A/B tests need careful design so that the two groups are comparable, and they must still respect every safety and privacy control, since real customers are involved.

Evaluation also has a cost. Model judges consume tokens, and human reviewers' time is expensive. Ravi budgets for evaluation as part of every project, typically running the full set before each release and a small daily sample in production.