Safety, misuse and responsible deployment

A language model released to millions of people will be asked everything: homework questions and medical worries, requests for help with code and with crimes, attempts to make it say something embarrassing, and instructions hidden in documents by attackers. Lesson 5 described how fine-tuning and reinforcement learning shape a model's behaviour, and Lesson 7 covered the security risks of connecting models to tools and data. This lesson takes the deployer's point of view. It surveys the risks that arise when language models are used at scale, the layers of safeguards that developers and deployers apply, jailbreaks and red-teaming, the balance between preventing harm and refusing legitimate requests, honesty and sycophancy, privacy and memorization, methods for marking AI-generated text, fairness, and the governance practices that hold the whole system together. The deeper research questions of alignment and control are the subject of AI-430.

Risks of deployed language models

The risks of a deployed model fall into several groups:

• Harmful assistance: providing meaningful help with weapons, cyberattacks, fraud or violence, or content that sexualizes minors.
• Misinformation and hallucination: stating falsehoods confidently, including fabricated citations, which is especially harmful in medicine, law and finance.
• Privacy: revealing personal information from training data or from a user's context to someone who should not see it.
• Manipulation and over-reliance: telling users what they want to hear, encouraging unhealthy dependence, or nudging opinions.
• Misuse at scale: generating spam, phishing messages, fake reviews or coordinated influence campaigns more cheaply than before.
• Bias and discrimination: treating people differently by group in ways that cause harm, particularly in consequential decisions.
• Unsafe actions: agents that send messages, spend money or modify systems doing so wrongly or under an attacker's control.

The relevant risks depend on the deployment. A coding assistant inside a company faces different risks from a public chatbot used by teenagers, so responsible deployment starts with an assessment of who will use the system, for what, and what could go wrong.

Layers of safeguards

As with fire protection in a building, no single measure is sufficient, so safeguards are layered:

• Model training. Safety training teaches the model to decline clearly harmful requests, to handle sensitive topics carefully, and to resist manipulation, using methods from Lesson 5 such as reinforcement learning from human and AI feedback guided by written principles.
• System instructions and policies. Deployers instruct the model about its role, audience and limits.
• Input and output classifiers. Separate models screen prompts and responses for categories of harm. Open classifiers such as Meta's Llama Guard are available to any deployer, and frontier developers use specialized classifiers for the highest-risk categories, such as the constitutional classifiers that Anthropic described in early 2025 for blocking requests related to chemical and biological weapons.
• Rate limits, identity checks and access tiers, which make large-scale misuse harder and allow riskier capabilities to be offered only to vetted users.
• Monitoring and enforcement. Abuse detection looks for patterns across many requests, such as an account systematically probing for weapons information, and leads to warnings or account termination under the provider's usage policies.
• Human review and escalation for flagged or high-stakes cases.

Each layer has a cost in latency, money or false refusals, and the strength of safeguards is matched to the severity of the risk.

Jailbreaks and red-teaming

A jailbreak is an input designed to make a model ignore its safety training. Techniques include role-play ("pretend you are an AI without rules"), framing a harmful request as fiction or research, splitting it across many innocuous steps, encoding it in another language or cipher, many-shot jailbreaking (filling a long context with fabricated examples of the model complying, a technique Anthropic published in 2024), and automatically optimized adversarial strings appended to a prompt, demonstrated by researchers in 2023 to transfer across models. Jailbreaks show that safety training produces tendencies rather than guarantees.

Red-teaming is the deliberate search for such failures before and after release. Internal teams, external domain experts (for example, in biology or cybersecurity), crowdsourced participants and automated attack systems try to elicit harmful behaviour, and the failures they find are used to improve training and classifiers. Many developers also run bug bounty programs that reward outside researchers for reporting universal jailbreaks. Because new jailbreaks keep appearing, safeguards must be monitored and updated continuously, and the most dangerous capabilities are protected by several independent layers so that one successful jailbreak is not enough.

The cost of refusing

Safeguards can fail in two directions. A system that misses harmful requests causes harm; a system that refuses legitimate requests fails its users, frustrates nurses asking about drug doses, security professionals asking about vulnerabilities and novelists writing about crime, and pushes them to less careful tools. Over-refusal was a common complaint about early chatbots, and developers now measure it as carefully as they measure harmful compliance.

The trade-off is clearest for classifiers. A classifier assigns each request a score, and the deployer chooses a threshold above which requests are blocked. Because harmful requests are rare, even a small false positive rate on benign traffic can mean that most blocked requests were legitimate.

import numpy as np

# Choosing a threshold for a harmful-request classifier: missed harm vs over-refusal
rng = np.random.default_rng(11)
benign = rng.beta(2, 8, 99_000)            # classifier scores for 99,000 benign requests
harmful = rng.beta(8, 2, 1_000)            # scores for 1,000 harmful requests (1% of traffic)

print("threshold  harmful blocked  benign refused  refusals that were benign")
for th in (0.5, 0.6, 0.7, 0.8, 0.9):
    tp = (harmful >= th).sum()
    fp = (benign >= th).sum()
    print(f"   {th:.1f}        {tp / len(harmful):6.1%}         {fp / len(benign):6.2%}          {fp / (tp + fp):6.1%}")

# Two-stage design: a cheap classifier flags, a stronger model or human reviews flagged cases
flag = 0.5
reviewer_accuracy = 0.98
flagged_h = (harmful >= flag).sum()
flagged_b = (benign >= flag).sum()
blocked_h = flagged_h * reviewer_accuracy
refused_b = flagged_b * (1 - reviewer_accuracy)
print(f"two-stage at {flag}: harmful blocked {blocked_h / len(harmful):.1%}, benign refused {refused_b / len(benign):.3%}, "
      f"reviews needed {flagged_h + flagged_b:,} of {len(benign) + len(harmful):,}")

threshold  harmful blocked  benign refused  refusals that were benign
   0.5         97.7%          1.90%           65.8%
   0.6         92.8%          0.37%           28.5%
   0.7         78.9%          0.05%            6.3%
   0.8         55.8%          0.00%            0.4%
   0.9         23.0%          0.00%            0.0%
two-stage at 0.5: harmful blocked 95.7%, benign refused 0.038%, reviews needed 2,856 of 100,000

At a threshold of 0.5, the classifier blocks 98 percent of harmful requests, but two-thirds of the requests it refuses are benign. Raising the threshold to 0.8 nearly eliminates false refusals but lets through almost half the harmful requests. A two-stage design, in which a cheap classifier flags about 3 percent of traffic for a more capable model or a human to review, blocks about 96 percent of harmful requests while refusing only a few benign requests in 10,000. The right operating point depends on the severity of the harm: for the most dangerous categories, deployers accept more false refusals; for minor harms, they favour helpfulness.

Honesty, hallucination and sycophancy

Language models generate plausible text, and plausible is not the same as true. Hallucinations arise because training rewards fluent continuations, and because evaluation practices that score only correct answers encourage models to guess rather than admit uncertainty. Mitigations include grounding answers in retrieved sources with citations (Lesson 7), training models to express uncertainty and to say "I don't know", verifying claims with tools, and designing interfaces that make sources easy to check.

Sycophancy is the tendency to tell users what they want to hear: agreeing with mistaken claims, praising weak work, or changing a correct answer when a user pushes back. It emerges partly from training on human preferences, because people tend to rate agreeable answers highly. In April 2025, OpenAI rolled back an update to its GPT-4o model after users reported that it had become excessively flattering and validating, including of harmful decisions, and published an explanation of how preference-based training signals had contributed. Measuring and correcting sycophancy is now a standard part of model evaluation.

Privacy and memorization

Models can memorize parts of their training data, particularly text that appears many times, and reproduce it when prompted. Research in 2021 extracted hundreds of verbatim training sequences from GPT-2, including names, phone numbers and addresses, and in 2023 researchers showed that asking ChatGPT to repeat a single word indefinitely could make it diverge and emit memorized training text, a problem OpenAI then mitigated. Deduplication (Lesson 4), filtering personal information from training data, and output filters reduce the risk; differential privacy, which adds noise during training to limit what can be learned about any individual example, gives formal guarantees but at a significant cost in quality for large models.

Deployment adds further privacy questions: what happens to the prompts and documents users submit, whether they are used for training, how long they are kept, and who can see them. Enterprise and API offerings typically exclude customer data from training and offer data retention controls, while consumer products vary. Applications built on language models must also enforce access controls on retrieved data, as Lesson 7 explained, so that the model cannot reveal one user's documents to another.

Marking AI-generated text

As generated text becomes indistinguishable from human writing, people want to know which is which: teachers checking assignments, platforms fighting spam, and readers judging sources. Post-hoc detectors, classifiers trained to recognize machine-written text, have proved unreliable. They are easy to evade by light editing and paraphrasing, and a 2023 study found that popular detectors misclassified essays by non-native English writers as AI-generated far more often than essays by native speakers. Decisions about students or employees should never rest on such detectors alone.

Watermarking embeds a detectable signal at generation time. In the green-list scheme proposed by researchers in 2023, the previous token is hashed to divide the vocabulary into a green list and a red list, and the model's probabilities are nudged toward green tokens. Human text contains green tokens at the base rate, while watermarked text contains significantly more, so a detector that knows the hashing key can compute a z-score:

where \(G\) is the number of green tokens among \(T\) tokens and \(\gamma\) the green fraction. Google DeepMind deployed a related method, SynthID-Text, in its Gemini products and released it as open source in 2024.

import numpy as np

# Green-list text watermark: generation prefers a pseudo-random "green" half of the vocabulary,
# chosen by hashing the previous token; detection counts green tokens and computes a z-score.
rng = np.random.default_rng(7)
V, gamma = 5000, 0.25                 # vocabulary size, fraction of vocabulary that is green

def is_green(prev, tok):
    return (prev * 7919 + tok * 104729) % 1000 < gamma * 1000

def generate(T, watermark, p_green_boost=0.5):
    toks = [int(rng.integers(V))]
    for _ in range(T):
        tok = int(rng.integers(V))
        if watermark and not is_green(toks[-1], tok) and rng.random() < p_green_boost:
            while not is_green(toks[-1], tok):       # resample until green (stands in for a logit bias)
                tok = int(rng.integers(V))
        toks.append(tok)
    return toks

def z_score(toks):
    T = len(toks) - 1
    g = sum(is_green(a, b) for a, b in zip(toks, toks[1:]))
    return (g - gamma * T) / np.sqrt(T * gamma * (1 - gamma)), g / T

def paraphrase(toks, frac):
    out = toks.copy()
    for i in rng.choice(range(1, len(out)), int(frac * (len(out) - 1)), replace=False):
        out[i] = int(rng.integers(V))
    return out

for T in (25, 100, 400):
    wm = generate(T, True)
    z1, f1 = z_score(wm)
    z0, f0 = z_score(generate(T, False))
    z2, f2 = z_score(paraphrase(wm, 0.4))
    print(f"{T:3d} tokens: watermarked z = {z1:5.1f} ({f1:.0%} green), human z = {z0:5.1f} ({f0:.0%}), "
          f"40% paraphrased z = {z2:4.1f}")
print("a z-score above 4 is a common detection threshold (false positive rate about 3 in 100,000)")

 25 tokens: watermarked z =   2.2 (44% green), human z =  -0.1 (24%), 40% paraphrased z =  2.2
100 tokens: watermarked z =   8.3 (61% green), human z =   0.0 (25%), 40% paraphrased z =  2.3
400 tokens: watermarked z =  19.1 (66% green), human z =   0.7 (26%), 40% paraphrased z =  5.3
a z-score above 4 is a common detection threshold (false positive rate about 3 in 100,000)

The output shows that detection needs enough text: 25 tokens give weak evidence, while a few hundred give overwhelming evidence. Paraphrasing 40 percent of the tokens weakens the signal sharply, although long texts remain detectable. Watermarks can also be removed by translating text or by having another, unwatermarked model rewrite it, and they work only for models whose operators choose to apply them, so they are one tool for provenance rather than a solution. For images, audio and video, cryptographically signed content credentials (Lesson 9) provide a complementary approach.

Fairness and consequential decisions

Language models absorb social biases from their training data. Bias appears in obvious forms, such as stereotyped descriptions, and in subtle ones: studies have found that models give different recommendations, salary estimates or evaluations of identical résumés depending on names associated with different genders or ethnic groups. Evaluation uses benchmarks of biased associations and, more usefully for a specific application, counterfactual testing, in which the same input is submitted with only a demographic attribute changed and the outputs are compared.

When language models contribute to consequential decisions about people (hiring, lending, insurance, housing, access to services), human rights and privacy law apply. In Canada, human rights codes prohibit discrimination regardless of whether a person or an algorithm makes the decision, and Quebec's Law 25 requires organizations to inform people when a decision about them is based exclusively on automated processing and to let them submit observations to a person who can review it. Keeping a qualified human accountable for such decisions, documenting how the system was tested, and monitoring outcomes by group are basic safeguards.

Governance

Responsible deployment is an organizational practice, not only a technical one. Developers publish model cards or system cards describing a model's capabilities, evaluations, limitations and safeguards; frontier developers publish safety frameworks that define capability thresholds and the protections required at each. Deployers maintain an inventory of the AI systems they use, assess each for risk before launch, assign owners, train staff, set up channels for users to report problems, log and investigate incidents, and review systems as models and uses change. Standards such as ISO/IEC 42001, an AI management system standard published in 2023, and the US NIST AI Risk Management Framework provide structures for these practices, and regulations such as the European Union's AI Act increasingly require them. AI-440 covers the legal and policy landscape in detail.