Jailbreaks, prompt injection and red teaming

Language models are trained to refuse harmful requests and to follow their developers' and users' instructions, but these behaviours are learned, not guaranteed, and adversaries search constantly for inputs that break them. A jailbreak persuades a model to produce content it was trained to refuse; a prompt injection hijacks a model's behaviour through instructions hidden in content it processes. As models become agents that read web pages, emails and documents and take actions with tools, these attacks move from embarrassing outputs to real-world consequences. AI-410, Lesson 11 introduced these risks and the layers of safeguards used in deployment. This lesson goes deeper into the attack landscape, the measurement of robustness, red teaming as a discipline, and the specific security problem of agents.

Families of jailbreaks

Jailbreak techniques evolve quickly, but they fall into recognizable families:

• Role-play and framing: asking the model to act as a character without restrictions, or embedding the request in fiction, hypotheticals or "research" framing.
• Obfuscation: encoding requests in other languages, ciphers, base64, misspellings or fragmented pieces that the model reassembles.
• Many-shot and context attacks: filling a long context with many fabricated examples of the model complying, exploiting in-context learning.
• Optimization-based attacks: algorithms that search for adversarial suffixes or prompts that maximize the probability of compliance, sometimes producing gibberish strings that transfer between models.
• Multi-turn escalation: gradually steering a conversation from benign to harmful over many turns, each step only slightly beyond the last.
• Fine-tuning attacks: for models whose weights or fine-tuning interfaces are available, a small amount of training on harmful examples can remove safety behaviours, which is why open-weight releases and fine-tuning APIs need separate risk assessment.

Understanding families helps defenders generalize: training against one specific prompt is easily bypassed, while training against a family of techniques builds broader resistance.

Prompt injection

Prompt injection exploits the fact that a language model receives instructions and data through the same channel: text. When an assistant summarizes a web page, reads an email or processes a document, any instructions embedded in that content may be followed as if they came from the user. In direct prompt injection, the user themselves supplies the malicious instructions; in indirect prompt injection, a third party plants them in content the model will later read, such as hidden text on a website telling a browsing agent to send the user's data to an attacker. For agents with access to email, files, payments or code execution, a successful injection can cause data exfiltration, unauthorized actions or the spread of the injection to other systems.

There is currently no complete defence against prompt injection, because distinguishing legitimate instructions from adversarial ones in natural language is fundamentally hard. Mitigations combine training models to treat content from tools as data rather than instructions, marking and separating trusted and untrusted inputs, classifiers that detect injection attempts, and above all system design that limits damage: least-privilege tool permissions, confirmation by the user before consequential actions, restricting where data can be sent, and isolating untrusted content (Lesson 10).

Measuring robustness

Robustness against jailbreaks and injections must be measured, and measurement has statistical and methodological pitfalls. The main metrics are the attack success rate (ASR), the fraction of attack attempts that achieve the attacker's goal, and, equally important, the over-refusal rate, the fraction of benign requests wrongly refused. A defence that cuts ASR but refuses many legitimate requests imposes real costs on users and may push them to less safe alternatives. Judging whether an attempt "succeeded" is itself difficult: graders, human or model-based, must decide whether the output actually provides the harmful capability, not just whether it failed to refuse.

import numpy as np
from scipy.stats import norm

def wilson(k, n, conf=0.95):
    z = norm.ppf(1 - (1 - conf) / 2)
    p = k / n
    mid = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return mid - half, mid + half

# Red-team results: harmful requests that succeeded, and benign requests wrongly refused
results = {
    "Baseline model":          dict(attacks=(118, 500), refusals=(14, 1000)),
    "Plus input/output filter": dict(attacks=(21, 500), refusals=(61, 1000)),
}
for name, r in results.items():
    k, n = r["attacks"]; lo, hi = wilson(k, n)
    k2, n2 = r["refusals"]; lo2, hi2 = wilson(k2, n2)
    print(f"{name:<25} attack success {k / n:5.1%} [{lo:5.1%}, {hi:5.1%}]   over-refusal {k2 / n2:4.1%} [{lo2:4.1%}, {hi2:4.1%}]")

# Repeated attempts: a low per-attempt success rate still adds up
for p in (0.001, 0.01, 0.04):
    for tries in (10, 100, 1000):
        print(f"per-attempt success {p:5.1%}, {tries:4d} attempts: P(at least one) = {1 - (1 - p) ** tries:6.1%}")

Baseline model            attack success 23.6% [20.1%, 27.5%]   over-refusal 1.4% [0.8%, 2.3%]
Plus input/output filter  attack success  4.2% [ 2.8%,  6.3%]   over-refusal 6.1% [4.8%, 7.8%]
per-attempt success  0.1%,   10 attempts: P(at least one) =   1.0%
per-attempt success  0.1%,  100 attempts: P(at least one) =   9.5%
per-attempt success  0.1%, 1000 attempts: P(at least one) =  63.2%
per-attempt success  1.0%,   10 attempts: P(at least one) =   9.6%
per-attempt success  1.0%,  100 attempts: P(at least one) =  63.4%
per-attempt success  1.0%, 1000 attempts: P(at least one) = 100.0%
per-attempt success  4.0%,   10 attempts: P(at least one) =  33.5%
per-attempt success  4.0%,  100 attempts: P(at least one) =  98.3%
per-attempt success  4.0%, 1000 attempts: P(at least one) = 100.0%

The filter reduces measured attack success from about 24 % to about 4 %, with confidence intervals that do not overlap, a clear improvement. But over-refusal rises from about 1.4 % to 6.1 %, also a significant change, which the deployment decision must weigh. The second part of the output shows why low per-attempt success rates are not enough against determined attackers: at 1 % per attempt, 100 automated attempts succeed at least once 63 % of the time, and at 4 % per attempt, success is almost certain. Studies of "best-of-N" jailbreaking, which simply samples many random variations of a prompt, have confirmed this scaling. Defences therefore also need rate limits, monitoring for repeated attempts and account-level enforcement.

Red teaming

Red teaming is the practice of deliberately attacking a system to find its failures before others do. For AI systems, it takes several forms:

• Internal red teams: staff dedicated to probing models for harmful capabilities and vulnerabilities, often organized by risk area.
• External experts: specialists in areas such as biosecurity, cybersecurity or child safety, who bring domain knowledge needed to recognize genuinely harmful outputs.
• Automated red teaming: using language models to generate large numbers of diverse attacks, mutate successful ones, and grade results, which scales testing but can miss creative human strategies.
• Public bug bounties and challenges: inviting outside researchers to find jailbreaks, with rewards, which has proven effective at discovering universal jailbreaks.

Good red teaming is structured: it starts from a threat model (who might attack, with what goals and skills), defines what counts as a successful attack, records attempts and results systematically, prioritizes findings by severity, and feeds them back into training, classifiers and system design. It is repeated whenever the model or system changes, because fixes can introduce new weaknesses.

Defence in depth for deployed models

Deployed systems combine several layers, so that an attack must defeat all of them: safety training of the model itself (including adversarial training on jailbreak families); input classifiers that screen prompts; output classifiers that screen responses, including streaming checks that can stop generation mid-response; system prompts and tool permissions that limit what the model can do; monitoring and rate limits that detect repeated or automated attempts; and incident response when a new jailbreak is found. Classifier-based safeguards trained on a written specification of prohibited content, sometimes called constitutional classifiers, have been shown in red-teaming programmes to block the large majority of universal jailbreak attempts at modest over-refusal cost, though no layer is perfect.

Bug bounties and continuous red teaming

A red-team campaign before release captures only the attacks that its participants think of in a limited time. After release, millions of users and many motivated researchers probe the system, and new attack techniques appear every few months. Developers therefore supplement pre-deployment campaigns with continuous measures:

• Bug bounty programs: paying external researchers for reproducible jailbreaks, especially universal ones that bypass safeguards across many harmful requests. Several developers have run public and invitation-only bounties focused on their most serious safeguards.
• Automated red teaming: using attacker models to generate, mutate and test prompts at scale, with classifiers judging success, so that safeguards are tested against thousands of variations each day.
• Regression suites: keeping every successful attack found so far in a test set that every new model and safeguard version must pass.
• Threat intelligence: monitoring forums, papers and abuse reports for new techniques and adding them to evaluations quickly.

The goal is to shorten the time between a new attack appearing and the system being defended against it, and to measure that time as a key indicator. A single campaign number, however carefully estimated, describes the system only on the day it was measured.

Red-team findings should also feed back into training: successful jailbreaks become training examples for refusal behaviour and classifiers, and patterns of injection inform the design of agent permissions and tool interfaces.