Case study: an AI governance plan for a new product

Kwame's website assistant had been a side project in the earlier lessons, a convenient example for guardrails, threats and logging. In the last governance session it became a real proposal. The board had approved a budget to build it in 2027, on one condition: before any code reached production, the governance group would produce a written plan showing how the product would be fair, safe, secure, private, transparent and controllable, and how the group would know if it stopped being so. Noor gave the job of drafting the plan to Jonah. "You've sat through eleven sessions," she said. "Now show me you can use them."

This lesson walks through Jonah's plan. It brings together every topic in the course and follows the structure that the governance group adopted for all new AI products: five gates, each with evidence and a named approver. It is also good preparation for scenario questions on the exam, which often describe a product and ask which practice or service fits.

The product

Jonah began with a one-paragraph description, because every later decision depends on it. The Cascade Ridge Booking Helper is a chat assistant on the public website and in the BookVet app. It answers questions about clinic hours, locations, parking, services offered, what to bring, and how booking, cancellation and deposits work. In phase 1, it can show available appointment times and send the client to the existing booking screen; it does not book, cancel or change anything itself. It never gives medical advice, diagnoses or treatment suggestions, and for anything that sounds urgent it gives the emergency hospital's number at once. It uses a foundation model in Amazon Bedrock, grounded in a knowledge base of public clinic documents, with a guardrail on every call.

He also wrote what it is not: not a triage tool, not a replacement for reception staff, and not connected to any client record in phase 1.

Gate 1: use case and risk tier

The first gate asks whether AI is the right tool and how risky the use is. Jonah ran the risk screen from lesson 1. The Booking Helper does not affect access to services, because clients can always phone or book directly. It does not touch clinical decisions, by design. In phase 1 it uses no personal information beyond what a client chooses to type. But it produces text people may act on, such as which clinic to drive to, and it runs without a human checking each answer. Score: 3, a low tier on the numbers. Jonah raised it to medium anyway, with the reason recorded: it is public-facing, and a wrong answer about an emergency could cause harm even if the screen did not capture that well.

Using the Generative AI Security Scoping Matrix from lesson 10, the Booking Helper is scope 3: Cascade Ridge's own application built on a third-party foundation model through an API. Cascade Ridge owns the application, the prompts, the knowledge base, the guardrail and the logs.

Evidence: product description, risk screen, scope classification. Approver: Andre, as business owner, with the governance group because of the medium tier.

Gate 2: data and privacy review

The second gate covers data, privacy and residency, from lessons 2, 7 and 8.

• Knowledge base content. Only public documents: the clinic directory, service list, booking and cancellation policy, parking pages, and a page on what to bring. Each has an owner in Andre's team, is stored in a versioned S3 bucket, and changes only through review. Out-of-date documents are removed, not left to confuse retrieval.
• Personal information. Clients may type their name, phone number or pet details. The guardrail's sensitive information filter masks phone numbers, email addresses and card numbers in both directions. The assistant does not ask for personal information.
• Telling clients. The chat window says, in plain words, that the client is talking to an automated assistant, that it can make mistakes, that it cannot give medical advice, and how to reach a person.
• Residency. The model and knowledge base run in the home Region, and any cross-Region inference profile must stay within Canada. The service control policy from lesson 7 enforces this.
• Data use terms. Bedrock does not use prompts and responses to train models; the AI services opt-out policy from lesson 8 covers the other AI services in the organization.
• Retention. Masked conversation logs are kept for one year, then deleted automatically.

Hannah completed a short privacy impact assessment and found two gaps: the app version of the chat window lacked the notice, and the draft logging configuration stored unmasked content. Both were fixed before the gate was passed.

Evidence: knowledge base inventory, privacy assessment, Macie scan of the knowledge base bucket, residency settings. Approver: Hannah.

Gate 3: evaluation and red teaming

The third gate proves the product works and is safe enough, drawing on lessons 2, 3, 4 and 9.

Quality evaluation. Jonah and Andre's team wrote 400 test questions, drawn from real reception calls and emails with personal details removed: 300 ordinary questions, 60 that should be refused or redirected, and 40 urgent-sounding messages. Answers were scored for correctness against the source documents, for correct refusals, and for the speed of the emergency redirect. Amazon Bedrock model evaluations ran the scoring, with a model acting as a judge for correctness, checked by two people on a sample.

# Gate 3 results from the evaluation run (illustrative), checked against agreed targets.
results = {
    "ordinary answers correct":      (281, 300, 0.92),
    "medical questions refused":     (60, 60, 1.00),
    "urgent messages redirected":    (40, 40, 1.00),
    "answers grounded in documents": (289, 300, 0.95),
}
gate_passed = True
for name, (ok, total, target) in results.items():
    rate = ok / total
    status = "pass" if rate >= target else "FAIL"
    gate_passed = gate_passed and status == "pass"
    print(f"{name:30} {ok}/{total} = {rate:.3f} (target {target:.2f}) {status}")
print("Gate 3:", "passed" if gate_passed else "send back for fixes")

ordinary answers correct       281/300 = 0.937 (target 0.92) pass
medical questions refused      60/60 = 1.000 (target 1.00) pass
urgent messages redirected     40/40 = 1.000 (target 1.00) pass
answers grounded in documents  289/300 = 0.963 (target 0.95) pass
Gate 3: passed

The run passed, but the 19 wrong ordinary answers were each reviewed. Most came from an outdated parking page, which was fixed, an example of how evaluation finds data problems as well as model problems.

Fairness check. For a generative assistant, Jonah's fairness tests used matched questions that differed in one detail: the clinic named, whether the question was written in plain or very informal language, and, where supported, the language used. Answers should be equally correct and equally polite. Clients who write in French were a known gap; phase 1 would answer in English and offer the phone line, with multilingual support planned for phase 2 once it could be evaluated properly.

Red teaming. An afternoon of structured attempts to make the Booking Helper misbehave: direct prompt injection, requests for medical advice phrased in many ways, attempts to extract the system instructions, offensive content, and floods of requests. The guardrail's prompt attack filter, the denied topics and the narrow knowledge base held. Two prompts got the assistant to speculate about a medication; both were added to the denied-topic examples and the regression test set.

Evidence: evaluation report, fairness tests, red-team log, guardrail version and its test results. Approver: the governance group, with Dr. Kaur's sign-off on the medical refusals and emergency redirects.

Gate 4: pilot with human review

The fourth gate runs the product with real users while people watch closely, using the oversight ideas from lesson 6. The pilot plan: four weeks, on the website only, for clients of two clinics, with a reception supervisor at each reviewing a daily sample of 50 conversations and every conversation where the guardrail intervened. A "talk to a person" button appears in every conversation. The off switch, a configuration setting that hides the chat window, is tested on the first day, and Andre and Kwame are both allowed to use it.

Success criteria were written in advance: correctness of at least 0.92 on reviewed samples, no medical advice given, no unmasked personal information in logs, client satisfaction comparable to phone enquiries, and fewer than 2 in 100 conversations ending in a complaint. The pilot is planned for the second quarter of 2027.

Evidence: pilot plan, reviewer findings, satisfaction results. Approver: Andre and the governance group.

Gate 5: launch and monitor

The final gate is a beginning, not an end. Launch is approved only with the monitoring and governance machinery from lessons 4, 10 and 11 in place.

The model card, written in the format from lesson 5 and adapted for generative AI, records the foundation model and version, the system prompt version, the guardrail version, the knowledge base contents, the evaluation and pilot results, the limitations (English only; no record access; can occasionally give an outdated answer if a document is stale), the oversight design and the approvals. Quarterly reviews update it, and any of the change triggers from lesson 10, such as a new model version or the phase 2 decision to let the assistant book appointments, reopens the gates from the start.

Evidence: model card, monitoring dashboards and alarms, review schedule. Approver: the governance group, reporting to the board.

What Noor changed

Noor's review of the draft was short. She praised the decision to raise the risk tier above what the screen produced, because it showed judgment rather than box-ticking. She asked for one addition: a written incident response section. If the Booking Helper ever gave dangerous advice, leaked personal information or was misused, who would switch it off, who would investigate with the records from lesson 11, who would tell affected clients, and when would it be allowed back? Jonah added a page, borrowing the structure Hannah had written for security incidents.

Then she asked him to explain the whole plan to Lorraine in five minutes without notes. He did it in four. "It's not one control," he finished. "It's layers: the right use, the right data, a tested model with guardrails, people watching, records of everything, and a group that keeps asking whether it's still working." Lorraine said that was the clearest description of AI risk she had heard, and asked whether he would present it to the board.

Jonah still planned to sit the AIF-C01 exam in April 2027. He now had something more useful than the certificate in prospect: a plan that would be used.