Safety cases and capstone project

Each lesson of this course has contributed a kind of evidence about whether an AI system is safe: alignment training and its limits, robustness testing, red teaming, interpretability, capability and propensity evaluations, monitoring and control, and governance processes. A decision to deploy a powerful model needs all of this evidence assembled into a coherent argument that can be examined, challenged and updated. That argument is a safety case. This final lesson explains what safety cases are, where the idea comes from, the main kinds of argument used for frontier AI, how to reason about layered defenses and their weaknesses, and how to recognize gaps. It then sets out a capstone project in which you build a safety case for a realistic deployment.

What a safety case is

A safety case is a structured argument, supported by evidence, that a system is acceptably safe for a specific use in a specific environment. The idea comes from safety-critical industries: nuclear power, aviation, rail, defense and medical devices. In the United Kingdom, for example, operators of nuclear installations must produce safety cases, and the ISO 26262 standard for road vehicles requires a safety case for safety-related electronic systems. Safety cases shift the burden of proof to the developer: rather than regulators searching for problems, the developer must show why the system is safe, and reviewers test the argument.

Safety cases are often written in a structured notation such as Goal Structuring Notation (GSN) or Claims, Arguments and Evidence (CAE). A top-level claim (for example, "deploying model M through product P does not pose unacceptable risk of serious harm through biological weapons uplift") is broken into sub-claims, each supported by arguments and, ultimately, evidence such as evaluation results, red-team reports or monitoring data. Assumptions and context are stated explicitly, and defeaters, reasons the argument might fail, are identified and addressed.

Kinds of argument for frontier AI

Researchers have proposed several families of argument that safety cases for advanced AI can use, roughly in order of how much they rely on understanding the model:

• Inability arguments: the model cannot cause the harm even if it tried, because capability evaluations show it lacks the relevant capability with a margin (Lesson 8). These are the strongest and most common arguments today, but they weaken as models improve and depend on good elicitation.
• Control arguments: even if the model has the capability and is misaligned, deployment safeguards and monitoring prevent it from causing harm, as shown by control evaluations against a red team (Lesson 10).
• Trustworthiness or alignment arguments: the model would not cause the harm even though it could, because alignment evaluations, audits and interpretability provide evidence that it is not pursuing harmful goals (Lessons 6, 7 and 9). These are currently the hardest to make convincingly.
• Deference arguments: for future systems, the judgment of trusted AI systems might itself support safety claims, though this requires establishing their trustworthiness first.

For misuse risks, a related structure applies: either the model cannot provide meaningful uplift, or safeguards (refusal training, classifiers, monitoring and access controls) reduce misuse to an acceptable level, demonstrated by red teaming and bounty programs. Several developers have published safety case sketches and use safety case reasoning internally when deciding whether models above capability thresholds can be deployed.

Layered defenses and correlated failures

Safety arguments for misuse often rely on several layers of defense, an approach sometimes called defense in depth or the Swiss cheese model: each layer has holes, but harm occurs only when holes line up across every layer. If layers fail independently, the probability that an attack passes all of them is the product of the miss rates of each layer, which can be very small. But layers are rarely independent. A novel kind of jailbreak may defeat refusal training and classifiers trained on similar data at the same time, and an attacker who is skilled enough to evade one layer may be skilled enough to evade others.

import numpy as np
# A layered safety argument. Each layer independently stops a given attack with some probability.
# The chance an attack gets through all layers is the product of the miss rates, if failures are independent.
layers = {"Refusal training": 0.90, "Input and output classifiers": 0.80, "Usage monitoring": 0.60, "Access controls": 0.50}
miss = np.array([1 - p for p in layers.values()])
print(f"Independent layers: attack gets through with probability {miss.prod():.4f}")
# Correlated failures: some attacks (for example a novel jailbreak style) defeat several layers together.
rng = np.random.default_rng(12)
for share_common in [0.0, 0.2, 0.5]:
    trials, through = 200000, 0
    common = rng.random(trials) < share_common          # attacks of a novel kind
    passed = np.ones(trials, bool)
    for p in layers.values():
        stop = rng.random(trials) < p
        stop[common] = rng.random(common.sum()) < p * 0.3  # layers much weaker against novel attacks
        passed &= ~stop
    print(f"Share of novel attacks {share_common:.0%}: attack gets through with probability {passed.mean():.4f}")

Independent layers: attack gets through with probability 0.0040
Share of novel attacks 0%: attack gets through with probability 0.0043
Share of novel attacks 20%: attack gets through with probability 0.0813
Share of novel attacks 50%: attack gets through with probability 0.1960

With four independent layers stopping 90 %, 80 %, 60 % and 50 % of attacks, an attack gets through only about 0.4 % of the time, and the simulation with no novel attacks agrees. But if 20 % of attacks are of a novel kind against which every layer is much weaker, the pass-through rate rises about twentyfold, to roughly 8 %; with half the attacks novel, it reaches about 20 %. A safety case that multiplies layer effectiveness as though failures were independent can overstate safety by orders of magnitude. Good safety cases therefore test the system end to end with red teams that attack all layers together, look for common causes of failure, and include layers that work in different ways (for example, access controls and monitoring that do not depend on recognizing the content of an attack).

Reviewing a safety case

A safety case is only as good as its weakest link, and reviewers look for typical gaps:

• Unsupported claims: sub-claims without evidence, or evidence that does not actually address the claim.
• Weak elicitation: inability claims based on evaluations that did not try hard enough to bring out capabilities.
• Unstated assumptions: for example, that users cannot fine-tune the model, that the model does not recognize evaluations, or that monitoring will not be circumvented.
• Scope drift: a safety case for one deployment (a chat product with classifiers) reused for another (an API with fine-tuning access, or open release).
• Staleness: evidence from an earlier model checkpoint, or from before new elicitation methods or attack techniques appeared.
• Missing defeaters: plausible reasons for failure that the argument has not considered.

Safety cases should be living documents, updated as models, deployments, attacks and evidence change, with explicit triggers for re-evaluation such as a new capability threshold being approached or a serious incident.

Capstone project

The capstone brings the course together. You will write a safety case, of about 3,000 to 5,000 words with supporting code, for a realistic deployment of an AI system. Choose one of the following scenarios or propose your own of similar scope:

• A customer-service agent for a bank that can read account information and initiate refunds up to a limit.
• A coding agent with access to a company's repositories and continuous integration systems.
• A general-purpose chat assistant released to the public, with a focus on misuse risks from dangerous technical knowledge.
• A research assistant for a biology laboratory with access to literature, databases and laboratory scheduling tools.

Your safety case should include the following parts:

• Scope: the system, deployment, users, permissions and environment, with assumptions stated explicitly.
• Hazard analysis: the harms that matter (misuse, misaligned actions, accidents, prompt injection, data leakage) and how they could occur.
• Top-level claim and structure: a claim broken into sub-claims in GSN or CAE style, with a diagram.
• Evidence: at least three kinds of evidence you produce or simulate yourself using methods from the course, for example a red-team evaluation with confidence intervals (Lesson 5), a capability evaluation with pass@k and contamination checks (Lesson 8), a calibration or sycophancy evaluation (Lesson 9), a control protocol simulation with safety and usefulness (Lesson 10), or a probe with a control task (Lesson 6).
• Defeaters and gaps: what could make the argument fail, including correlated failures, and how you address them or why they remain open.
• Governance: decision process, monitoring, incident response and the triggers for revisiting the safety case, and which laws and frameworks apply (Lesson 11).
• Conclusion: whether the deployment should go ahead, with which conditions, and what further evidence would most strengthen the case.

For governance, explain which US rules apply (for example federal sector regulators and SR 11-7 model risk guidance for a bank, state privacy laws, and California SB 53 or the New York RAISE Act if the underlying model is a frontier model), and which foreign rules, such as the EU AI Act, would apply if the system were offered in Europe.

Your project will be assessed on the clarity and rigor of the argument, the quality and honesty of the evidence (including its limitations), the identification of realistic defeaters, and the quality of the recommendation. A safety case that concludes the deployment should not proceed, or should proceed only with significant restrictions, is a perfectly good outcome if the evidence supports it.