Risks, harms and content safety

The final session of the course, on December 23, is the one Maya has been preparing for: a dry run of the presentation she will give the AI review group on January 6. She has the impact assessment from the last lesson, the test results so far and a draft design. What she needs now is a clear account of what could go wrong with a generative AI assistant, how bad each problem could be, and what the company will do about it. Arjun suggests organizing it the way Microsoft organizes its own guidance for generative AI: identify the potential harms, measure them, mitigate them in layers, and operate the system responsibly once it is live.

Identify: what can go wrong

Generative AI brings some risks that are familiar from other software and some that are new. Arjun and Maya list the ones that matter for a customer service assistant, grouped by who could be harmed.

Identifying harms is creative work, and it benefits from many viewpoints. Arjun runs a short red-teaming exercise: Maya's most experienced agents spend an afternoon trying to make the test assistant misbehave, using the strangest, rudest and most cunning messages they have received over the years. Red teaming, the practice of deliberately attacking a system to find its weaknesses before real users do, finds three problems the developers had not imagined, including a customer persona that persuaded the assistant to "make an exception just this once".

Measure: how often and how badly

Each identified harm needs a way to measure it, so that the team can tell whether a mitigation works. Measurement combines a test set of prompts designed to provoke each harm, automated scoring and human review. Foundry's evaluation tools can score responses for qualities such as groundedness, relevance and coherence, and for safety risks such as harmful content and jailbreak success, using models as graders alongside human judgment.

# Red-team results before and after adding mitigations (counts out of 120 attempts per harm)
before = {"fabricated policy": 19, "dangerous advice": 7, "jailbreak success": 11, "privacy leak": 2}
after  = {"fabricated policy": 3,  "dangerous advice": 0, "jailbreak success": 2,  "privacy leak": 0}
for harm in before:
    b, a = before[harm], after[harm]
    print(f"{harm:20s} before {b:3d}  after {a:3d}  rate after {a / 120:.3f}")

fabricated policy    before  19  after   3  rate after 0.025
dangerous advice     before   7  after   0  rate after 0.000
jailbreak success    before  11  after   2  rate after 0.017
privacy leak         before   2  after   0  rate after 0.000

The numbers become the evidence in Maya's presentation. They also set a baseline for monitoring after launch, because a sudden rise in any of them would mean something has changed.

Mitigate: layers of protection

No single control stops every harm, so Microsoft recommends mitigations at several layers, each catching some of what the others miss.

• The model layer: choose a model suited to the task, and a provider that has trained it to refuse harmful requests. Check its documentation and test it.
• The safety system layer: platform-level filters that inspect every prompt and response. In Azure this is Azure AI Content Safety, available through Foundry, described below.
• The system message and grounding layer: instructions that define scope and refusals, plus grounding in trusted documents, which together reduce fabrication and off-topic behavior.
• The user experience layer: interface design that sets expectations and keeps people in control, such as AI disclosure, source links, limits on input length, clear feedback buttons and an easy route to a person.

Content safety in Azure

Azure AI Content Safety detects harmful content in text and images. Its core classifiers cover four categories: hate, sexual, violence and self-harm. For each category it assigns a severity level, and a deployment's content filter can be configured to block content at or above a chosen level. Models deployed in Foundry have a default content filter applied to both prompts and responses, and organizations can adjust its thresholds within the limits Microsoft allows for their use case. Content Safety also offers more specialized checks: prompt shields, which detect attempts to override a model's instructions, including indirect attacks hidden in documents; groundedness detection, which checks whether a response is supported by the supplied sources; and protected material detection, which flags known copyrighted text and code.

# Applying a content filter: block a category when its severity meets the threshold.
# Severity uses a 0 to 7 scale here; thresholds are illustrative.
thresholds = {"hate": 2, "sexual": 2, "violence": 4, "self_harm": 2}
def decide(scores):
    blocked = [c for c, s in scores.items() if s >= thresholds[c]]
    return ("blocked: " + ", ".join(blocked)) if blocked else "allowed"
samples = {
    "complaint with mild swearing": {"hate": 0, "sexual": 0, "violence": 0, "self_harm": 0},
    "question about safe use of an axe": {"hate": 0, "sexual": 0, "violence": 2, "self_harm": 0},
    "abusive message about a group": {"hate": 4, "sexual": 0, "violence": 1, "self_harm": 0},
}
for name, scores in samples.items():
    print(f"{name:35s} -> {decide(scores)}")

complaint with mild swearing        -> allowed
question about safe use of an axe   -> allowed
abusive message about a group       -> blocked: hate

The sample shows why thresholds need thought. A hardware store gets many legitimate questions involving knives, saws and axes, so a violence threshold set too low would block ordinary product questions. Some messages also touch on self-harm, and Carmen insists that the assistant respond to any such message with care: a short, kind reply that directs the person to 988, the Suicide and Crisis Lifeline, and an immediate handoff to a trained agent, rather than a cold refusal or an attempt to continue the conversation.

Operate: after launch

Responsible operation starts before launch and continues for as long as the system runs. Before launch: a phased rollout, starting with a small share of chats, and a documented plan for incidents, including how to switch the assistant off. After launch: monitoring of quality and safety measures, review of samples of real conversations, a simple way for customers and agents to report problems, and regular retesting whenever the model, the prompts or the documents change. Models are updated and retired by their providers, so a deployment that worked in January may need retesting in June. Sunil adds security monitoring: unusual volumes of requests, repeated jailbreak attempts and calls to tools at odd hours should all raise alerts.

Rehearsing an incident

Arjun insists on one more exercise before the dry run ends: a rehearsal of something going wrong. The scenario is that on a Saturday morning a customer posts a screenshot online showing Ask Finch promising a full refund on a cut-to-size countertop, which the policy excludes. Who notices, and how? Who decides what to do? Within ten minutes the group has answers. The social media team flags the post to Maya, who is on the weekend call list. She can switch Ask Finch to a "handoff only" mode, in which every chat goes straight to an agent, from a setting in the application, without waiting for a developer. Arjun pulls the conversation from the logs to see which sources were retrieved and why the answer went wrong. Dana decides that the company will honor the promise to that customer, because a mistake by the company's assistant is the company's mistake. And the team adds the case to the test set so that the fix can be checked, and checked again after every future change.

The rehearsal exposes one gap: the logs show the prompt and the answer, but not which version of the system message was in use at the time. Arjun adds the version number to every log entry. Maya notices that the whole exercise echoes the principles of the previous lesson, with accountability, transparency and reliability turned into concrete steps that someone actually performs on a Saturday morning.

Overreliance and the agents' drafts

The last harm on the list, overreliance, applies to staff as well as customers. When Ask Finch drafts replies for agents to send, there is a risk that busy agents will send them without reading carefully, especially after weeks of drafts that were nearly always right. Mitigations here are mostly in the user experience: drafts are clearly marked as drafts, the source passages are shown beside them, the send button is a separate step from accepting the draft, and the weekly review checks how often drafts are sent unchanged. If that number approaches every draft, it is a sign of rubber-stamping rather than of a perfect assistant.

The decision

Maya finishes the dry run with a one-page summary. The pilot will start in late January 2027 with one chat in ten, during staffed hours only, in English and Spanish. Ask Finch will answer returns, delivery and store-hours questions from grounded sources, will refuse electrical, gas, plumbing and structural advice, cannot see payment details, cannot change orders and will offer a person in every conversation. Content filters stay at the default thresholds except violence, which is raised one level after testing showed too many tool questions being blocked. Quality and safety measures will be reviewed weekly, and the pilot will stop automatically if fabricated-policy answers exceed one in a hundred in the weekly sample.

Dana asks the last question: "What does it cost if we're wrong?" Maya has the answer ready, because she has been asked a version of it every week since November. The pilot is small, every conversation has a human exit, and the assistant cannot do anything irreversible. The worst realistic failure is an annoyed customer and an embarrassing screenshot, and the company has a plan for both. Arjun smiles. In January, MSAZ-111 will follow the team as they build the pilot in Microsoft Foundry, and Maya will sit AI-901 in February.