Model cards and documentation

Hannah Weiss had a simple test for any system that touched client data. She would imagine that, a year from now, a client complained, a journalist called, or an auditor arrived, and that the person who built the system had left. Could someone else answer the questions: what is this, what data does it use, how well does it work, who approved it, and what are its limits? For the no-show model, the honest answer in November was "partly, if Noor is in the room".

"Then we write it down," Noor said, and opened a template. This lesson covers the documentation that makes AI systems understandable and accountable: model cards, data documentation, AI Service Cards and provider documentation, and the registry that keeps track of model versions. The exam guide names Amazon SageMaker Model Cards as a tool for transparency and for documenting data origins, and expects you to know why documentation matters.

Why documentation matters

Documentation turns knowledge in one person's head into a shared record. For responsible AI it does several jobs at once.

• Transparency. People who use or are affected by a model can learn what it is for and how well it works.
• Accountability. The record shows who decided what, when and why, including the trade-offs from lessons 2 and 3.
• Safe reuse. A model built for one purpose is often tempting to reuse for another. Documented intended uses and limits make it clear when that would be inappropriate.
• Audit and compliance. Regulators, auditors and insurers increasingly ask for evidence of how AI systems were built and tested. A model card answers many of their questions in one place.
• Continuity. When people change roles, the next owner can maintain the model without guessing.

Lorraine added a practical point: the cyber insurance renewal questionnaire that Hannah had filled in during the autumn had included two new questions about AI use. A complete model card would make next year's answers easy.

What a model card is

A model card is a short, structured document that describes a machine learning model: what it does, how it was built, how well it performs, and how it should and should not be used. The idea was proposed by researchers in 2019 as a standard way to report models, much as a nutrition label reports food. It is now widely used across the industry.

Noor's card for the no-show model had seven sections. The overview gave the name, version, owner (Andre), builder (Noor), status and the risk tier from the governance screen. Intended uses said the score suggests which bookings might benefit from an extra reminder or a waitlist offer, and stated plainly that it must not be used to refuse a booking, charge a deposit or judge a client. Training data listed the sources, the date range, the labelling fix and the known gap for the newer rural clinics. Evaluation gave the overall results and the results for each clinic group, with the fairness measures. Limitations described where the model performs worst, such as new clients with no history. Human oversight described who sees the score and how staff override it. Approvals recorded the governance group's sign-off and the next quarterly review date.

Amazon SageMaker Model Cards

Amazon SageMaker Model Cards is a feature of SageMaker AI for creating and storing model cards in a consistent format alongside the model itself. A card can record the model's intended uses, risk rating, training details, evaluation results and additional information, and SageMaker can fill in some training details automatically from the training job. Cards can be exported as PDF documents to share with people who do not use AWS, and they have their own version history, so changes are tracked.

Model cards work with the SageMaker Model Registry, which keeps a catalog of model versions with their approval status, such as pending, approved or rejected. Noor's process linked the two: a model version could only be marked approved in the registry once its card was complete and signed off. That way, the card was not paperwork done after the fact, but a gate on the way to production.

aws sagemaker create-model-card \
    --model-card-name crvp-no-show-v3 \
    --model-card-status Draft \
    --content file://no-show-v3-card.json

# A completeness check Noor runs before a card can move to approval.
card = {
    "intended_uses": "Suggest bookings that may benefit from an extra reminder.",
    "out_of_scope_uses": "Refusing bookings, deposits, judging clients.",
    "training_data": "BookVet and VetLedger bookings, Oct 2023 to Sep 2026.",
    "evaluation_overall": "recall 0.62, precision 0.41",
    "evaluation_by_group": "",
    "limitations": "Weak for new clients with no booking history.",
    "human_oversight": "Staff may ignore or override any score.",
    "risk_rating": "low",
}
missing = [k for k, v in card.items() if not v.strip()]
print("ready for approval" if not missing else "missing: " + ", ".join(missing))

missing: evaluation_by_group

The check caught exactly what it was designed to catch: the results by group had not yet been pasted in. A card without subgroup results would hide the very gap lesson 2 had found.

Documenting data origins

The exam guide connects documentation with source citation and documenting data origins, using the terms data lineage and data cataloging. A model is only as trustworthy as its data, so the documentation must trace where the data came from.

• Data lineage records how data moved and changed: which source tables a dataset came from, which transformations were applied, and which model versions were trained on which dataset version. Lesson 11 covers lineage tools in detail.
• Data cataloging records what datasets exist, what they contain, who owns them and how they may be used. The AWS Glue Data Catalog stores technical information about datasets, such as their schemas and locations, and other services build business catalogs on top of it.
• Source citation in generative AI means recording where content came from: which documents a knowledge base was built from, and, at run time, which passages an answer used. The call-note summarizer's links back to the transcript are a form of source citation.

Noor also recorded the legal basis for using each source. The booking data was used for running clinic services, which clients had been told about when they registered. Hannah confirmed that predicting no-shows to send an extra reminder fit that purpose. A different use, such as marketing, would have needed a fresh look.

Documentation from AWS and model providers

Cascade Ridge did not build every model it used. For the summarizer, it would use a foundation model through Amazon Bedrock. In that case the documentation comes partly from others.

• AWS AI Service Cards describe some of AWS's own AI services, such as Amazon Transcribe and Amazon Rekognition, including their intended uses, limitations, design choices and best practices for responsible use. Noor read the Transcribe card before planning to use it on call recordings.
• Model provider documentation describes a foundation model's training approach, capabilities, known limitations and license terms. Each model available in Amazon Bedrock comes with documentation and an end-user license agreement from its provider.
• Licenses state what you may do with a model and its outputs. Open source models come with licenses that vary widely in what they allow, including commercial use.

Cascade Ridge's own documentation then adds what only Cascade Ridge knows: why it chose this model, what prompts and guardrails it applies, how it evaluated the model on its own call transcripts, and how outputs are reviewed. The provider's card and the customer's card together form the full picture, which mirrors the shared responsibility model from lesson 1.

A card for a generative AI system

The no-show model is a classic machine learning model, and the classic card fits it well. The call-note summarizer is different: Cascade Ridge does not train the underlying foundation model, and its behaviour depends heavily on the instructions, the retrieved context and the guardrail wrapped around it. Noor adapted the card for this kind of system, and her changes are a useful pattern.

First, the card identifies the foundation model and version used, and links to the provider's documentation and license. If the provider releases a new version, switching to it is treated as a change that needs testing, not a silent upgrade. Second, the card records the system prompt and its version, because changing a few words of instruction can change the output as much as changing the model. Third, it records the guardrail version applied. Fourth, the evaluation section describes how the summaries were tested on Cascade Ridge's own material: 300 past call transcripts, with consent and with client details removed, each summary scored by two emergency veterinarians for accuracy, missing information and invented information. Fifth, the limitations section lists what the testing found, such as weaker summaries when two people speak at once on the call.

Together these items describe the whole system, not just the model. Jonah wrote in his notes that for generative AI the card documents the model, the prompt, the guardrail and the data that ground it, because all four shape the result.

Documentation and the law

Documentation is increasingly expected by law and by contracts, not only by good practice. Canadian privacy law requires organizations to be accountable for the personal information they control and to be able to explain their practices; a record of what data a model uses and why is the simplest way to meet that duty. Customers and partners also ask. A large corporate client of the emergency hospital, which sends its security dogs there, had begun asking suppliers whether they used AI on its data. With a model card and a data catalog entry, Hannah could answer in a day instead of a month.

Keeping documentation alive

A model card written once and never updated becomes misleading. Noor set three rules. Every new model version gets an updated card before approval. Every quarterly review updates the evaluation section with production results. And any incident, such as a complaint or an error that reached a client, is added to the limitations section with what was done about it.

Jonah asked how long the card should be. Noor's answer was a page or two for most models, written for a reader who is intelligent but not a data scientist. Long technical appendices can be attached, but the card itself must be readable by Dr. Kaur, Lorraine or an auditor in ten minutes.