Case study: choosing an AI approach for a business problem

On November 10, 2026, Noor and Jonah presented the first results of the AI assessment to the Cascade Ridge leadership team: Lorraine Beaudry, Dr. Simone Kaur, Andre Leclerc, Hannah Weiss and Gwen Halvorsen. The question in front of them was concrete. Missed appointments were costing the clinics empty slots every day. Should the company use machine learning to reduce them, and if so, how? This lesson follows the recommendation from problem to pilot plan, using every idea from the course.

Step 1: the business problem and its measure

Jonah opened with the numbers. Over the six months to September 2026, 6 of every 100 appointments were missed, about 160 a day across the group. Each empty slot is lost revenue and a staff member waiting. The goal was to reduce misses by at least 1 in every 100 appointments within six months of the pilot, without refusing, charging or otherwise penalizing any client. Stating the goal this way gave the team a measure of success and a constraint before any model was discussed.

Step 2: is AI the right tool?

Noor ran the idea through the screen from lesson 11. There is no fixed rule that decides who will miss an appointment; the old rule of thumb caught only about 1 in 3 misses. A probabilistic answer is acceptable, because the action is a friendly extra reminder, not a decision about care or access. There are three years of labeled history, about 410,000 appointments. The value of rescued slots should exceed the cost of reminders, as the lesson 5 calculation suggested. And the risk of harm is low, provided the score is never used to refuse or downgrade a booking. The idea passed every question.

Step 3: which kind of AI?

The data is structured and tabular, the target is a clear yes or no, the front desk needs a short reason with each flag, and the scores must be cheap, fast and consistent every night. Each of those points to traditional supervised ML, specifically binary classification, rather than a foundation model. Noor also considered whether a managed AI service could do the job; none predicts no-shows from a company's own booking data, so a custom model trained on Cascade Ridge's history was needed. She had prototyped it in SageMaker Canvas, then rebuilt the best configuration as a repeatable SageMaker AI training job: a gradient-boosted tree with depth limited to avoid overfitting.

Step 4: the data

The training data combined BookVet and VetLedger records. Hannah's privacy rules were applied first: names, phone numbers, addresses and payment details were removed, booking numbers were replaced with random identifiers, and postal code was left out as a possible proxy for income. Fields filled in only after an appointment, such as checkout time and invoice total, were removed to prevent leakage. Missing reminder-confirmation values became an explicit "unknown". The data was split by time: 25 months for training, 5 for validation and the 6 most recent months as the test set.

Step 5: evaluation

On the test set of 60,000 appointments, the model caught 62 of every 100 misses (recall 0.62), and about 41 of every 100 flagged bookings were real misses (precision 0.41). Its AUC was 0.81. Accuracy was reported only beside the majority-class baseline, to show why it was not the measure that mattered. Training and validation recall differed by only 0.04, so the model was not overfitting. Recall by clinic ranged from 0.55 to 0.68; the lowest clinic's gap traced to incomplete reminder records, which Andre agreed to fix. Clarify showed the three strongest features were past miss count, days booked ahead and reminder confirmation, which matched the front-desk staff's own experience.

Step 6: how the model will be used

Every evening at 6 p.m., a batch job scores the next day's bookings and writes a list for each clinic. Staff send flagged clients an extra text or a call, voiced by Amazon Polly for clients who prefer calls, and offer slots that clients release to the waitlist. The list shows a short reason, such as "booked 5 weeks ahead; reminder not confirmed". Staff can ignore any flag. The score is never shown to clients and never used to refuse, double-book or charge.

Step 7: operations and governance

Gwen's production-readiness note from lesson 8 covered the rest. The pipeline is defined in SageMaker Pipelines and kept in source control. The approved version is recorded in the Model Registry, with a model card describing its intended use, data, evaluation and limits. Model Monitor checks input drift weekly, and recall and precision are recomputed weekly once outcomes are known. Input checks stop the job on bad data, and the old three-week rule runs as a fallback if scoring fails. All resources sit in the ML account in us-west-2, with least-privilege roles, KMS encryption and CloudTrail logging. Noor owns the model until June 2027; after that, Jonah does, with Gwen's team supporting the infrastructure.

Step 8: the pilot plan

The weakest number in the business case was how many caught misses a reminder actually rescues. Rather than guess, the team designed the pilot to measure it. From February to April 2027, six clinics will use the model's list; the other five will continue with the old rule. Clinics were assigned so that both groups include a mix of large and small, urban and rural. In May 2027, the team will compare the change in no-show rates between the two groups and decide whether to roll out, adjust or stop. Everything after November 10, 2026 is a plan, and the board approved it on that basis.

# Sizing the pilot: how many appointments, and what change could we hope to see?
pilot_clinics_per_day = 1450        # bookings per day at the six pilot clinics (illustrative)
days = 89                            # February to April 2027
baseline_rate = 0.06                 # 6 misses in every 100 appointments
recall = 0.62                        # share of misses the model flags
rescued_share = 0.25                 # assumption to be measured: flagged misses rescued

appointments = pilot_clinics_per_day * days
expected_misses = appointments * baseline_rate
rescued = expected_misses * recall * rescued_share
new_rate = (expected_misses - rescued) / appointments
print("pilot appointments:", appointments)
print("misses expected without change:", round(expected_misses))
print("misses rescued if the assumption holds:", round(rescued))
print("no-show rate would fall from", baseline_rate, "to", round(new_rate, 4))
print("reduction per 100 appointments:", round((baseline_rate - new_rate) * 100, 2))

pilot appointments: 129050
misses expected without change: 7743
misses rescued if the assumption holds: 1200
no-show rate would fall from 0.06 to 0.0507
reduction per 100 appointments: 0.93

If one in four flagged misses is rescued, the no-show rate would fall from 6.0 to about 5.1 per 100 appointments, close to the goal of 1 fewer in every 100. If the rescued share turns out lower, the goal will not be met, which is exactly why the pilot measures it rather than assuming it. Lorraine liked that the decision in May would rest on evidence from Cascade Ridge's own clinics.

The other ideas

The leadership team also agreed the next steps for the rest of the list. After-hours call summaries, a generative AI task, would be explored in a separate project with Dr. Kaur reviewing every summary, the subject of AWS-112 and AWS-113. Survey themes would use Amazon Comprehend immediately, since no custom model is needed. The radiograph-quality idea would wait until technicians had labeled a sample with Ground Truth. Vaccine due dates, invoice calculation and new-clinic prediction were closed as not suitable for ML. A governance plan for all AI work, covering fairness, oversight, security and regulation, would follow, which AWS-114 develops.

What the case teaches about the exam

Jonah noticed that almost every step of the meeting matched a task statement in Domain 1 of the AIF-C01 exam guide. Defining AI, ML and deep learning, and placing generative and agentic AI among them, came from lessons 1 and 2. Recognizing data types, preparing data, splitting it and choosing an inference type came from lessons 3 and 4. Choosing metrics, both technical and business, and diagnosing fit came from lessons 5 and 6. The lifecycle, the sources of models, managed versus self-hosted use and MLOps came from lessons 7 and 8. Matching tasks to SageMaker AI capabilities and managed services came from lessons 9 and 10. And deciding when ML is the wrong tool, and when traditional ML beats a foundation model, came from lesson 11. Exam questions in this domain are often small versions of this meeting: a short scenario, then a question about which technique, metric, inference type or service fits. Reading each scenario for the business goal, the kind of data and the cost of each error usually points to the answer.

What success and failure would look like

Before the pilot starts, the team wrote down how it would read the results, so that nobody could reinterpret them afterwards. Success means the no-show rate at the pilot clinics falls by at least 1 in every 100 appointments more than at the comparison clinics, with no rise in client complaints and no front desk reporting that the list is unmanageable. A partial result, a smaller fall, would lead to adjusting the threshold or the reminder method and running a second period. Failure, no measurable difference, would mean keeping the old rule and recording what was learned. Writing these criteria in advance is a simple guard against seeing what one hopes to see.

Jonah, who will own the model after June 2027, added one more line to the plan: a check every quarter, even if nothing seems wrong, that the list is still used, the reasons still make sense to staff, and the rate of misses is still being reduced. Models rarely fail dramatically; they fade quietly when nobody is looking.

The meeting ended with a short list of actions, each with an owner and a date: Andre to fix reminder recording at the lowest-recall clinic by the end of November; Kwame and Mateo to build the nightly job and the front-desk list in December and January; Hannah to update the privacy notice wording and approve access; Noor to finalize the model card and monitoring; and Jonah to prepare the comparison report template for May. Everything on the list is a plan as of November 10, 2026.