Explainability and transparency

Priya, a receptionist at the Lethbridge clinic, was one of the first staff members to see the no-show model's output in a test screen. Next to one booking was a score of 0.27 and a small flag. She asked the question every user eventually asks: "Why this one?" The client had been coming to the clinic for nine years. If the screen could not answer her, Priya said, she would not trust it, and she would not use it.

Noor took Priya's question to the next governance session. "She's right," Noor said. "People need to know when AI is being used, what it is for, and why it says what it says. If they don't, they either ignore it or trust it too much, and both are bad." This lesson covers the two ideas behind Priya's question: transparency, which is about openness regarding a system and its use, and explainability, which is about understanding why it produced a particular output. It also covers the trade-offs between them and other goals, the tools AWS offers, and the principles of designing explanations for real people.

Transparency and explainability are different

The two words are often used together, but they answer different questions.

A system can be transparent without being explainable: Cascade Ridge could publish exactly how the summarizer works and still not be able to say why a particular summary used a particular word. And a model can be explainable in the narrow sense, with feature contributions available, while its use is hidden from the people it affects. Responsible design aims for both.

Transparent models and opaque models

The exam guide asks you to describe the difference between models that are transparent and explainable and models that are not. The key idea is a spectrum.

At one end are interpretable models, sometimes called transparent or white-box models. A scorecard, a linear regression or a small decision tree can be read directly: you can see each factor and how much it contributes. If a small tree says "flag if the client missed two of their last five visits and booked more than 30 days ahead", anyone can follow it.

At the other end are opaque or black-box models: large ensembles of trees, deep neural networks and foundation models with billions of parameters. Their internal reasoning cannot be read by a person. These models can capture complex patterns and are often more accurate, but explaining them requires extra tools that approximate their behaviour from the outside.

Openness about a model is a related but separate idea. The exam guide mentions open source models, data and licensing among the tools for transparency. A model whose weights, training code and training data description are published can be inspected and tested by anyone, and its license states what you may do with it. A proprietary model offered through an API may be just as capable, but you rely on the provider's documentation for how it was built.

Explaining predictions from opaque models

The no-show model was a gradient-boosted tree, in the middle of the spectrum: too complex to read directly, simple enough to explain well. Noor used a technique based on Shapley values, often shortened to SHAP. The idea comes from game theory: treat each feature as a player in a game whose payout is the prediction, and work out each player's fair share of the difference between this prediction and the average prediction.

For Priya's booking, the explanation was clear. The average booking scored 0.07. Two missed visits in the last five pushed the score up by 0.14; booking 41 days ahead added 0.06; an unconfirmed reminder text added 0.05; a routine vaccine visit added 0.02; and nine years as a client pulled it down by 0.07. Together they came to 0.27. Priya read it and nodded. "So it's the two missed visits. That's fair. She had a sick parent last spring." The explanation did not just justify the score. It gave Priya information to use her own judgment.

# Feature contributions add up from the baseline to the prediction.
baseline = 0.07
contributions = {
    "missed 2 of last 5 visits": 0.14,
    "booked 41 days ahead": 0.06,
    "reminder text not confirmed": 0.05,
    "routine vaccine visit": 0.02,
    "client for 9 years": -0.07,
}
score = baseline + sum(contributions.values())
print("score:", round(score, 2))
top = sorted(contributions.items(), key=lambda kv: abs(kv[1]), reverse=True)[:3]
for name, value in top:
    direction = "raises" if value > 0 else "lowers"
    print(f"- {name} {direction} the score by {abs(value):.2f}")

score: 0.27
- missed 2 of last 5 visits raises the score by 0.14
- client for 9 years lowers the score by 0.07
- booked 41 days ahead raises the score by 0.06

Explanations come at two levels. A global explanation describes the model as a whole: which features matter most across all predictions. A local explanation describes one prediction, like Priya's. Both are useful. The global view helped Noor notice that postal code mattered little, which supported removing it, while the local view is what users need day to day. Other techniques include partial dependence plots, which show how the prediction changes as one feature varies, and counterfactual explanations, which say what would need to change for a different outcome, such as "if the reminder had been confirmed, the score would fall below the threshold".

Explaining generative AI outputs

Foundation models are the hardest to explain internally. Practical transparency for generative AI therefore focuses on things around the model. The most useful is source attribution: when a system answers from retrieved documents, it can show which passages it used, so a reader can check them. The call-note summarizer was designed to link each line of its summary to the place in the call transcript it came from. A veterinarian reading "owner reports vomiting twice since 6 p.m." could click through to the words the caller actually said.

Other techniques include asking the model to state its uncertainty, showing the instructions and guardrails the system applies, and evaluating the model with standard tests so its strengths and weaknesses are documented. The exam guide names Amazon Bedrock model evaluations as a tool here: they compare models on tasks and measures such as accuracy, robustness and toxicity, using automatic scoring, a model acting as a judge, or human reviewers, and the results become part of the system's documentation.

Trade-offs: performance, interpretability and safety

The exam guide asks you to identify trade-offs between model safety and transparency, and to measure interpretability against performance. Three trade-offs come up most often.

• Interpretability against accuracy. A simpler model may be slightly less accurate. For the no-show model, Noor tested a small tree that was easy to read; its recall was 0.55 against 0.62. The governance group judged that the explanations available for the boosted model were good enough, and kept the more accurate model. For a high-stakes decision the balance might go the other way.
• Transparency against safety and security. Publishing every detail of how a system's filters work can help someone get around them. A system might disclose that guardrails exist and what they are for, without publishing the exact words and thresholds.
• Transparency against privacy. Explaining a prediction can reveal information about the people in the training data, or about the person themselves to someone who should not see it. Priya can see why a booking was flagged; a client in the waiting room glancing at her screen should not.

These are judgments, not formulas. Noor's rule was to write each trade-off down, with the reason, in the model's documentation.

Human-centered explanations

An explanation that people cannot understand is not an explanation. The exam guide refers to human-centered design for explainable AI, including user-feedback mechanisms and AI decision transparency. Noor and Kwame, who was adding the score to the booking screen, used five principles.

• Fit the explanation to the person. Priya needs the top three reasons in plain words, not a chart of 40 features. Noor needs the full detail.
• Say what the output is for. The screen says the score suggests who might benefit from an extra reminder. It does not say who is a bad client.
• Show uncertainty honestly. A score of 0.27 is not a prediction that the client will miss. The screen shows "about 1 in 4 bookings like this are missed".
• Let people disagree. A "this looks wrong" button sends feedback to Noor, with the reason. Feedback becomes evidence for the next review.
• Disclose AI use. Staff know which screens use a model. Clients are told, in plain language, when an automated system contacts them.

Transparency obligations to clients vary with the law and the use. In Canada, private-sector privacy law requires organizations to be open about their practices, and Quebec's privacy law adds specific notice requirements when a decision is made exclusively by automated processing; Cascade Ridge operates in Alberta and British Columbia, but Hannah treats clear disclosure as good practice everywhere.

Explanations as a quality tool

Explanations are not only for users. Noor used them as a debugging tool, and this is one of the most practical reasons to build them in from the start. When she first ran Shapley explanations across the whole test set, one feature stood out: the booking's appointment slot number. It mattered far more than it should have. On investigation, the slot numbers had been reassigned at one clinic in the middle of the training period, and the model had learned a pattern that was really an artifact of the change. Without the explanation, the model would have looked fine on paper and failed quietly when the slots were reassigned again.

Explanations also help when a decision is questioned. If a client asked why they had received three reminder calls when their neighbour received one, Cascade Ridge could look up the stored explanation for each booking and give a truthful answer. Hannah added a rule to the design: the top factors for each score are saved alongside the score, so the explanation shown on the day can be reproduced later even if the model has since been retrained. Lesson 11 returns to this idea under logging and lineage.

Finally, explanations support oversight. A reviewer who sees why a model produced an output can spot errors a reviewer looking only at the output would miss. That is one reason the exam treats explainability and human review as partners: each makes the other more effective.