Human oversight and review

Dr. Simone Kaur had one condition for the call-note summarizer, and she stated it at every meeting. "A veterinarian reads the call notes before acting on them. Always. The AI saves us typing. It does not decide anything." The rule had been agreed in the autumn assessment: AI may assist, but every clinical decision, including diagnosis, treatment and triage priority, stays with a veterinarian or a registered veterinary technologist.

Noor agreed completely, and then asked a harder question. "How do we make sure the reading actually happens, at four in the morning, on the fortieth call of the night, when the summaries have been right for three weeks?" Keeping a human in the loop is easy to write in a policy and hard to make work in practice. This lesson covers the forms human oversight takes, how to route outputs to people sensibly, the tools AWS provides, and the traps that make oversight fail.

Why human oversight matters

Human oversight is the ability of people to monitor an AI system, understand its outputs, and intervene: correct an output, override a decision, or stop the system. It is the practical side of the controllability dimension from lesson 1. It matters for several reasons.

• Models make mistakes. Every model has an error rate. Oversight catches errors before they cause harm, especially in high-stakes cases.
• Accountability needs a person. A decision that affects someone should have a person who answers for it. Canadian professional rules for veterinarians place responsibility for clinical judgment on the licensed professional, whatever tools they use.
• Context the model lacks. Priya knew her client had a sick parent; the no-show model did not. People bring knowledge no dataset holds.
• Learning. Reviewers' corrections become new labels and evidence that improve the next version.

Levels of human involvement

Oversight is not one thing. It is useful to distinguish three levels.

The right level depends on risk. The no-show reminders are low risk, reversible and beneficial, so a human on the loop is enough. Clinical summaries feed decisions about animals' health, so a human in the loop is required. The governance group recorded the chosen level in each model card.

Routing outputs to people

Reviewing every output is not always practical, and it is not always necessary. A common pattern is to route outputs by risk and by confidence. Outputs that are high risk, or where the model is unsure, go to a person. Others take an automatic path, with a random sample still reviewed so that problems on the automatic path are not missed.

For Cascade Ridge's later idea of reading lab-report PDFs with Amazon Textract, Noor sketched exactly this design. Textract returns a confidence score for each field it extracts. A result value extracted with high confidence goes straight into a draft for the veterinarian to confirm, while a value with low confidence, such as a smudged number, is highlighted for a technician to check against the original before anyone relies on it.

# Route extracted lab values: low confidence or critical fields go to review.
CRITICAL = {"potassium", "glucose"}
THRESHOLD = 0.90
extracted = [
    ("glucose", 6.1, 0.97),
    ("potassium", 7.4, 0.82),
    ("albumin", 31.0, 0.95),
    ("alt", 65.0, 0.71),
]
for name, value, conf in extracted:
    needs_review = conf < THRESHOLD or name in CRITICAL
    route = "REVIEW" if needs_review else "auto-draft"
    print(f"{name:9} {value:6.1f}  confidence {conf:.2f}  -> {route}")

glucose      6.1  confidence 0.97  -> REVIEW
potassium    7.4  confidence 0.82  -> REVIEW
albumin     31.0  confidence 0.95  -> auto-draft
alt         65.0  confidence 0.71  -> REVIEW

Two choices are visible in the code. The threshold sets how much goes to review; lowering it sends less to people and risks more errors getting through. And critical fields always go to review, whatever the confidence, because the cost of an error there is high. Choosing both is a governance decision, informed by testing.

Amazon Augmented AI

Amazon Augmented AI (Amazon A2I) is an AWS service for building human review workflows into machine learning applications. You define the conditions under which an output should go to a person, such as a confidence below a threshold or a random sample, and A2I sends those items to a work team for review through a simple web interface. The reviewers' decisions come back to the application and can be stored for audit and used as new labels.

A2I has built-in integrations with Amazon Textract, for document extraction, and Amazon Rekognition, for image moderation, and it can be used with any custom model. Work teams can be your own staff (a private workforce), a vendor you choose, or, for non-sensitive tasks, a public crowd workforce through Amazon Mechanical Turk. Hannah ruled out the public option immediately: any task involving client or patient information would use Cascade Ridge's own staff only.

For the call-note summarizer, the review happens inside the clinical record system itself, not in a separate tool. The summary appears as a draft. The veterinarian reads it next to the transcript links, edits it, and confirms it. Only confirmed summaries are saved. The design detail that mattered most was that confirming required an action; there was no path for an unread draft to become part of the record.

Evaluation with human reviewers

Human review is also part of evaluating models before launch. Amazon Bedrock model evaluations can use human reviewers, either your own team or a team managed by AWS, to rate model outputs on criteria you define, such as accuracy, completeness or style. Noor used this approach for the summarizer: two emergency veterinarians rated summaries of past calls, and their ratings became the evaluation section of the model card. Amazon SageMaker Ground Truth offers similar human labelling and review workflows for building training data and evaluating models.

When oversight fails

Noor spent the rest of the session on the ways human oversight goes wrong, because a review step that exists only on paper is worse than none: it creates false confidence.

• Automation bias. People tend to accept a system's output, especially after it has been right many times. Reviewers start skimming, then approving without reading.
• Volume. If every output goes to review, reviewers are overwhelmed and quality drops. Routing by risk and confidence keeps the volume manageable.
• Missing information. A reviewer who sees only the output, without the source, cannot check it. The summarizer shows transcript links for this reason.
• No authority. If reviewers are discouraged from overriding the system, or overriding takes much longer than accepting, review becomes a formality.
• No feedback. If corrections disappear, the model never improves and nobody notices patterns of error.

The countermeasures follow from the problems. Dr. Kaur's team agreed to three. Each month, reviewers would do a short practice set of past summaries, some with deliberate errors planted in them, kept entirely separate from real patient records; the rate at which errors were caught would be tracked without blame, to see whether attention was slipping. The summary screen would highlight anything the model marked as uncertain, such as unclear medication names. And every edit a veterinarian made would be stored and reviewed monthly for patterns. If a type of error kept appearing, the prompt or guardrail would be changed.

Oversight for AI agents

Kwame's longer-term idea for the website assistant was to let it actually book appointments, not just answer questions. That would make it an agent: a system that uses a model to decide on actions and then carries them out through tools, such as the BookVet booking interface. Agents raise the stakes for oversight, because an error is no longer just a wrong sentence on a screen; it is a wrong action in a real system.

Noor's guidance for agents followed the same risk-based logic. Actions that are low risk and easy to undo, such as holding a slot for ten minutes, can run on their own, with monitoring. Actions that are harder to undo or that affect money or clinical care, such as cancelling another client's appointment or booking surgery, need a person to approve them before they happen. The agent should ask the client to confirm in plain words what it is about to do, and the clinic should be able to see and reverse anything an agent has done. Lesson 7 covers the security controls that limit what an agent's tools can do, and lesson 9 covers the threat of an agent being manipulated into taking actions it should not.

Recording human decisions

Oversight produces its own evidence. Each time a reviewer confirms, edits or rejects an AI output, the system records who did it, when, what the AI proposed and what the final version was. These records serve three purposes. They show auditors and the governance group that review really happens. They give Noor the corrected examples she needs to improve the model. And if a decision is ever questioned, they show which parts came from the AI and which from a person.

Hannah set one boundary on these records: they contain clinical and personal information, so they are stored with the same protection as the clinical record itself, kept only as long as the retention policy allows, and accessible only to the people who need them. Lesson 11 returns to logging and retention.

Designing for override and stop

Oversight also means being able to stop the system. Each AI feature at Cascade Ridge got a documented off switch: a configuration setting that turns the feature off and falls back to the old way of working, with a named person allowed to use it. For the summarizer, the fallback was simply typing notes by hand, as before. Andre tested the switch for the reminder job during the pilot planning, and it took under a minute. "Controllability you haven't tested," Noor said, "is a hope, not a control."