Case study: choosing a generative AI approach

By late January 2027 Jules has a decision to make. The steering committee has approved budget for one more generative AI project in the spring, and three proposals compete for it. The help desk wants an assistant that drafts replies to customer emails. The buying team wants a tool that compares supplier specification sheets. And the stores want a voice assistant on tablets that answers staff questions while they work the floor. Jules must recommend one, with an approach, a model choice, an evaluation plan and the controls it needs. This capstone lesson walks through his reasoning, pulling together every lesson in the course into a method that also works for scenario questions on the exam.

A method for choosing an approach

Ravi suggests a sequence of seven questions. Each one draws on earlier lessons, and together they turn a vague idea into a design.

• 1. What is the business problem and how will we measure success? Name the outcome and the KPI before naming any technology (lesson 9).
• 2. Is generative AI the right tool? Does the task need new content, work with unstructured data, or benefit from flexible language understanding? Would rules, search or a predictive model do better (lesson 1)?
• 3. What data does it need, and is that data good enough? Identify first-party, third-party and world data, and check quality, permissions and privacy (lessons 6 and 10).
• 4. How far up the adaptation ladder must we climb? Start with prompting; add grounding for facts, tools for live data and actions, and tuning only for behavior that prompts cannot achieve (lessons 5 to 8).
• 5. Which model fits? Choose by modality, measured quality, context, latency, cost, customization and governance (lessons 2 to 4).
• 6. How will we evaluate and monitor it? Build the evaluation set, set launch criteria and plan monitoring (lesson 9).
• 7. What could go wrong, and what controls are needed? Map limitations and risks to practices, decide the level of human involvement and assign an owner (lessons 10 and 11).

Proposal 1: drafting replies to customer emails

Problem and KPI. The help desk receives about 20,000 emails a month. Agents spend most of their time on routine questions about orders, returns and products. The goal is to cut the average handling time per email while keeping customer satisfaction at least as high. KPIs: handling time, satisfaction scores on replies, and the share of drafts sent with little editing.

Is generative AI right? Yes. Replies are new text, emails are unstructured, and wording varies endlessly, which rules cannot handle well. A predictive classifier already sorts emails into queues; the drafting adds the generative part.

Data. First-party: the policy documents, product catalog and order status system. Past emails and approved replies are valuable for examples and evaluation but contain personal information, so they must be redacted before use. Data quality work already done for the policy assistant helps.

Adaptation. Prompting with Birchline's tone rules and a few example replies; grounding in the policy documents through retrieval; a read-only order-status function so drafts can include real delivery dates. No tuning at first. Crucially, the system drafts and an agent sends: a human in the loop for every reply.

Model. A capable mid-sized model is a reasonable starting point, tested against a larger one on the evaluation set. Latency matters less than for live chat, because agents review drafts in a queue.

Evaluation. 300 past emails with redacted personal details and the replies agents actually sent; automated checks that cited policies exist and order details match the order system; a model judge for groundedness and tone, calibrated against ratings from senior agents.

Risks and controls. Hallucinated policies (grounding, citations, review); privacy (redaction, access controls on order data); automation bias (watch the edit rate and the share of drafts sent unchanged); prompt injection in incoming emails (screening, and drafts can only be sent by a person).

Proposal 2: comparing supplier specification sheets

Problem and KPI. Buyers compare specification sheets from competing suppliers, often scanned PDFs with tables in different layouts. Comparing three suppliers for a new tent takes a buyer most of a day. KPI: time to produce a comparison, and the accuracy of extracted fields checked against the source sheets.

Is generative AI right? Partly. Extracting fields from messy documents in many layouts is a strong multimodal task (lesson 3). The comparison itself, which supplier offers better value, is a business judgment that stays with the buyer.

Data. Third-party supplier documents, used within the supplier agreements. Volume is modest: perhaps a few hundred sheets a season.

Adaptation. A multimodal model with a structured output schema for the fields buyers need: fabric, weight, dimensions, warranty, certifications and price breaks. Low temperature. Validation in code for units and ranges, and a side-by-side table that shows each value with a link to its place in the source document. No agent or tuning needed.

Model. A capable multimodal model, because document understanding quality matters more than cost at this volume.

Evaluation. 50 sheets with fields extracted by hand; field-level accuracy per supplier format; special attention to units, which are a known source of error.

Risks and controls. Misread values (validation, source links, buyer review); confidentiality of supplier pricing (access limited to the buying team). Low risk overall, because a buyer checks every comparison.

Proposal 3: a voice assistant for store staff

Problem and KPI. Store associates often need answers while helping a customer: a policy detail, a product comparison, whether another store has stock. A hands-free voice assistant on tablets could answer in seconds. KPIs: questions answered without calling the help desk, time to answer, and associate satisfaction.

Is generative AI right? Yes for understanding spoken questions and answering from policies and product data. But the environment is hard: noisy stores, varied accents, and the need for very fast answers.

Data. First-party policies and catalog, already prepared for the policy assistant, plus live inventory through a function call. Voice recordings may capture customers nearby, which raises privacy questions under the applicable state privacy laws.

Adaptation. Speech recognition, retrieval over policies, a read-only stock function, and short spoken answers. Possibly a small, fast model for latency, or a small open model on the tablet for the stores with poor internet (lesson 3), with harder questions sent to a larger model in the cloud.

Model. Latency is the deciding factor, which favors smaller models and careful testing in real stores.

Evaluation. Recordings of staff questions made in a test store with background noise; speech recognition accuracy by accent and noise level; answer groundedness and speed.

Risks and controls. Misheard questions in noisy stores (confirmation for anything consequential); bias in speech recognition across accents (testing across the staff population); privacy of recorded customers (on-device processing where possible, no retention of audio, clear notices to staff). Higher complexity and more unknowns.

Comparing the proposals

Jules scores the three proposals on value, feasibility, risk and readiness of the data, using a simple weighted model the steering committee can see and question.

weights = {"value": 0.35, "feasibility": 0.25, "low_risk": 0.20, "data_ready": 0.20}
# Scores from 1 (poor) to 5 (strong), agreed in a workshop with each team.
proposals = {
    "email reply drafts":      {"value": 5, "feasibility": 4, "low_risk": 3, "data_ready": 4},
    "spec sheet comparison":   {"value": 3, "feasibility": 5, "low_risk": 5, "data_ready": 3},
    "store voice assistant":   {"value": 4, "feasibility": 2, "low_risk": 2, "data_ready": 3},
}
ranked = sorted(proposals, key=lambda p: sum(weights[k] * proposals[p][k] for k in weights), reverse=True)
for p in ranked:
    score = sum(weights[k] * proposals[p][k] for k in weights)
    print(f"{score:.2f}  {p}")

4.15  email reply drafts
3.90  spec sheet comparison
2.90  store voice assistant

The scores are only as good as the judgments behind them, and Jules says so in his recommendation. But the structure makes the reasoning visible. The email drafting assistant offers the largest value, reuses the grounding work already done, keeps a person in control of every reply and has clear KPIs. The specification comparison is low risk and easy, so Jules proposes it as a small side project for the buying team using the same platform. The voice assistant is promising but needs a field trial in two stores before any commitment.

The recommendation

Jules's final recommendation fits on one page, the format the board liked in January.

• Build: the email reply drafting assistant, launching to one help desk team in spring 2027 and expanding if KPIs are met.
• Approach: prompting with tone rules and examples, grounding in policy documents, a read-only order-status function, human approval of every reply. No tuning in the first release.
• Model: a mid-sized model chosen by evaluation against a larger one; revisit a tuned small model if volume grows.
• Evaluation: 300 redacted past emails, automated checks, a calibrated model judge and senior-agent ratings; launch only if groundedness and edit-rate targets are met.
• Controls: redaction and access control, screening for prompt injection, monitoring of the share of drafts sent unchanged, and Jules as business owner with Teresa approving privacy and security.
• Also: the specification comparison as a small project for buyers; a two-store field trial of the voice assistant before any further investment.

Ravi reads it and has only one comment: "You did not mention a single model by name until step five." Jules takes that as the compliment it is. The model is one decision among seven, and rarely the most important one.

Applying the method on the exam

Scenario questions on the Generative AI Leader exam often describe an organization, a goal and some constraints, then ask for the best approach, product or next step. The same method helps. Identify the business goal first. Look for words that point to a technique: "current" or "our own documents" suggests grounding; "consistent style" suggests tuning after prompting; "take action" or "look up live data" suggests agents and tools; "images" or "documents" suggests multimodal models. Watch for constraints such as data residency, limited budget, low latency or regulatory scrutiny. Prefer the simplest approach that meets the need, keep people in the loop where stakes are high, and remember that responsible AI is part of every correct answer, not an optional extra. GCP-191 practices these patterns in depth.

Checking the recommendation

Before sending the recommendation, Jules asks Teresa and Dana to try to break it. Teresa asks what happens if a customer's email contains another customer's order number; the order-status function must check that the order belongs to the sender. Dana asks what the assistant will cost if volume doubles at the holidays; Ravi's estimate shows it stays well within budget with the mid-sized model. Both answers go into the recommendation, which is stronger for having been challenged.