Capabilities and limitations of generative AI

Dr. Simone Kaur, the medical director, came to the fifth AI hour with a printout of twenty overnight call summaries that Jonah had produced in a test using anonymized old notes. She had marked them with a red pen. Fifteen were, in her words, "better than what a tired technician writes at four in the morning". Three were fine but wordy. Two had problems: one turned "the dog may have eaten a sock" into "the dog ate a sock", and one left out that the owner had already given an antacid.

"So it is good," she said, "until it is not, and it does not tell you which is which." Noor said that was the best one-sentence summary of generative AI she had heard. This lesson sets out what generative AI does well, where it falls short, and how to weigh the two for a business problem, using the advantages and disadvantages named in the AWS Certified AI Practitioner exam guide.

Advantages

The exam guide names several advantages. Each showed up in Jonah's test.

• Adaptability. One model handles many tasks with no retraining: summarizing call notes, drafting a client letter, classifying a complaint, extracting fields from a referral. Changing the task means changing the instructions, not building a new model.
• Responsiveness. Answers arrive in seconds, so a summary can be ready by the time the morning team starts its handover.
• Conversational capability. People use the model in plain language and can ask follow-up questions, so no special training or query language is needed. Dr. Kaur asked the test system to "list only the cases that might come back today" and it did.
• Content generation. The model produces first drafts, rewrites, summaries and variations quickly, which saves staff time on writing.

Other strengths follow from these. Models work across many languages, which matters for letters in French. They handle unstructured text that older systems struggled with, such as free-form notes full of abbreviations. They can be put to work quickly, because an organization can use a provider's model instead of collecting training data and building its own.

Disadvantages

The guide names four disadvantages specifically: hallucinations, limited interpretability, inaccuracy and nondeterminism. Dr. Kaur's red pen had found examples of three of them.

• Hallucination. The model states something that is not supported by its input or by fact, fluently and confidently. Turning "may have eaten a sock" into "ate a sock" is a small hallucination with real clinical weight. Lesson 6 is devoted to it.
• Inaccuracy. Even without inventing anything, the model can be wrong: leaving out the antacid, misreading a dose, or summarizing the less important half of a note. Accuracy varies by task, by model and by how the input is written.
• Limited interpretability. It is hard to explain why a model produced a particular output. A transformer's billions of parameters do not map to reasons a person can read, so you cannot point to the rule that caused a mistake the way you could with a written procedure.
• Nondeterminism. The same input can produce different outputs, as lesson 3 explained. Two runs of the same note might mention the antacid once and miss it once, which makes testing harder: one good result does not prove the next one will be good.

Noor added further limitations that the exam covers elsewhere or that every business meets in practice.

• Knowledge cutoff and missing private knowledge. The model knows nothing after its training date and nothing about the organization's own records unless they are supplied.
• Bias. Models learn from human-written data and can reproduce stereotypes or treat groups unequally. The responsible AI course in this program covers this in depth.
• Context limits. Very long inputs may not fit, and important details in the middle of a long input can be overlooked.
• Prompt injection and misuse. Text inside the input, such as a client message, can try to override the model's instructions. Applications need guardrails, covered in later courses.
• Cost and latency at scale. A few cents a request is cheap until it is a million requests; large models are also slower.
• Intellectual property and data questions. Who owns generated content, whether it resembles copyrighted material, and where input data goes all need policy answers before wide use.

Judging the fit for a business problem

Advantages and limitations matter only in relation to a specific task. Noor gave Jonah four questions to ask about any proposed use.

Applied to the overnight call summaries, the answers were mostly encouraging. The summaries are reviewed by the morning vet, so errors can be caught. Many phrasings are acceptable. The needed knowledge is in the note itself. And Dr. Kaur had just shown that quality can be scored. The two flagged problems pointed to a design rule rather than a reason to stop: the summary must keep the caller's hedges, such as "may have" and "possibly", and must list every medication mentioned. The full note stays one click away.

Applied to the rejected idea of letting clients ask an AI about symptoms, the answers were all warning signs: no review step, clinical consequences, and no way for the model to examine the animal. That idea stayed rejected.

Measuring instead of guessing

Because outputs vary and errors are not flagged by the model itself, the only reliable way to judge a generative AI system is to test it on realistic examples and count the results. Noor turned Dr. Kaur's red-pen review into a simple scoring sheet, and Jonah wrote a few lines of Python to tally it.

# Dr. Kaur's review of 20 test summaries, one label each.
reviews = (["good"] * 15) + (["wordy"] * 3) + ["hedge lost", "medication missed"]
serious = {"hedge lost", "medication missed"}

total = len(reviews)
usable = sum(1 for r in reviews if r in ("good", "wordy"))
problems = sum(1 for r in reviews if r in serious)
print("summaries reviewed:", total)
print("usable as written:", usable, "of", total)
print("serious problems:", problems, "of", total)
print("serious problems per 100 summaries:", round(100 * problems / total))

summaries reviewed: 20
usable as written: 18 of 20
serious problems: 2 of 20
serious problems per 100 summaries: 10

Ten serious problems per hundred summaries was far too many to rely on without review, but it was a baseline. After the next round of changes, the same twenty notes plus thirty new ones would be scored the same way. Noor pointed out that the number would never reach zero, which is exactly why the vet review step had to stay. The question for the pilot was whether summaries plus review saved time compared with reading every full note, and whether the review reliably caught the misses.

# Nondeterminism check: the same note summarized five times (outputs recorded by hand).
runs = [
    "Beagle ate raisins ~2 h before call; no vomiting; owner en route.",
    "Beagle, raisin ingestion about 2 h ago, not vomiting, coming in now.",
    "Beagle ate raisins; owner bringing in now.",
    "Beagle ate raisins ~2 h ago; no vomiting yet; en route.",
    "Raisin ingestion in beagle ~2 h prior; no vomiting; owner on the way.",
]
must_mention = ["raisin", "vomit", "2 h"]
for i, r in enumerate(runs, 1):
    missing = [m for m in must_mention if m not in r.lower()]
    print(i, "OK" if not missing else "missing: " + ", ".join(missing))

1 OK
2 OK
3 missing: vomit, 2 h
4 OK
5 OK

Run three dropped two key facts even though the input never changed. That is nondeterminism in practice, and it is why the team tests each change on many examples, not one.

Capabilities improve; the review principle stays

Hannah asked whether a newer, bigger model would make the problems go away. Noor's answer was measured. Newer models make fewer errors on many tasks, handle longer inputs and follow instructions better. None of them guarantees correctness, and all of them can still hallucinate. The pattern the exam rewards, and the one Cascade Ridge adopted, is to use generative AI where it saves time on drafting and summarizing, design the workflow so a qualified person checks anything consequential, and measure quality continually rather than assuming it.

Where generative AI adds the most value

Looking across the use cases Cascade Ridge had considered, Noor pointed out a pattern that matches what many organizations report. Generative AI tends to add the most value where three conditions meet: there is a lot of unstructured text that people currently read or write by hand; the output is a draft or a summary that a person will use rather than a final decision; and an error, if it happens, will be noticed before it causes harm. Overnight call summaries, answers from the procedure manual and first drafts of client letters all fit. Invoice amounts, drug doses and triage priorities do not.

A second pattern concerns who benefits. The largest time savings often go to experienced staff, who can check a draft quickly because they know what a good one looks like. New staff gain help too, but they are also least able to spot an error. That is one reason Dr. Kaur insisted that summaries be reviewed by the morning vet rather than by a new technician, and that training for new staff include examples of summaries that went wrong.

Limitations that are easy to miss

Some limitations show up only after a system has been in use for a while. Model providers update and retire versions, so behaviour can change. Staff may begin to rely on a tool for tasks it was never tested on, a pattern called scope creep; someone might start pasting lab reports into the summary tool simply because it is convenient. And outputs written in a confident, polished style can carry more authority than they deserve, especially when they come from a tool the organization has approved. The group's response was procedural rather than technical: a short list of approved uses for each tool, a request route for new uses, and periodic checks of how tools are actually being used. None of these limitations is a reason to avoid generative AI. Each is a reason to manage it like any other system that affects clients and patients.

Jonah summarized the lesson for his own study notes in a way that also suits the exam. Know the four advantages and the four disadvantages by name. For any scenario, ask what happens when the output is wrong and who will notice. If the answer is that nobody will notice until a client or patient is affected, the scenario is describing a poor fit, or a design that needs a review step before it can be used.