Robustness and adversarial examples

A system that behaves well on its test set may behave badly when conditions change or when someone deliberately tries to make it fail. Robustness is the property of performing reliably across distribution shifts, unusual inputs and adversarial pressure. It matters for safety in two ways: unreliable behaviour on unusual inputs can cause accidents, and deliberate attacks can turn a well-intentioned system to harmful ends. This lesson covers distribution shift and out-of-distribution detection, adversarial examples and why they exist, the fast gradient sign method and stronger attacks, defences including adversarial training and certified robustness, data poisoning and backdoors, and what robustness means for large language models.

Distribution shift

Models learn patterns from training data and are evaluated on data from the same distribution. In deployment, the distribution shifts: new users, new environments, new kinds of requests. A medical imaging model trained on one hospital's scanners may fail on another's; a self-driving perception system trained in sunny weather may fail in snow; a language model trained before an event knows nothing of it. Shifts can be gradual or sudden, and they can reveal shortcuts the model learned, such as relying on a hospital's watermark rather than the pathology. Defences include training on diverse data, evaluating on deliberately shifted test sets, monitoring input distributions in deployment (AI-420, Lesson 7), and out-of-distribution detection: flagging inputs unlike the training data so that they can be handled cautiously, for example by deferring to a human. Well-calibrated uncertainty (Lesson 9) is valuable here, because a model that knows when it does not know can avoid confident mistakes.

Adversarial examples

In 2013, Szegedy and colleagues showed that image classifiers could be fooled by adding tiny, carefully chosen perturbations to images, changes imperceptible to humans that caused confident misclassification. These adversarial examples exist for almost every kind of model, including language models, speech recognizers and reinforcement learning agents. One explanation, offered by Goodfellow and colleagues, is linearity in high dimensions: if a model is approximately linear in its input, a small change to each of thousands of input dimensions, each aligned with the sign of the model's weights, adds up to a large change in the output. Another view is that models rely on non-robust features: patterns in the data that are genuinely predictive but imperceptible or meaningless to humans, which an attacker can manipulate.

The fast gradient sign method (FGSM) creates an adversarial example in one step, moving each input feature by a small amount ε in the direction that increases the loss:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression

# Fast gradient sign method (FGSM) attacks on linear digit classifiers with different regularization
digits = load_digits()
X = digits.data / 16.0                      # pixel intensities scaled to 0 to 1
y = digits.target
Xtr, ytr, Xte, yte = X[:1300], y[:1300], X[1300:], y[1300:]

def fgsm(model, X, y, eps):
    # input gradient of the cross-entropy loss for multinomial logistic regression
    p = model.predict_proba(X)
    grad = (p - np.eye(10)[y]) @ model.coef_
    return np.clip(X + eps * np.sign(grad), 0, 1)

print("C (weaker regularization = larger C)   clean   eps 0.05   eps 0.10   eps 0.20")
for C in (10, 1, 0.1, 0.01):
    m = LogisticRegression(max_iter=5000, C=C).fit(Xtr, ytr)
    accs = [m.score(Xte, yte)] + [m.score(fgsm(m, Xte, yte, e), yte) for e in (0.05, 0.1, 0.2)]
    print(f"C = {C:<5}                              " + "    ".join(f"{a:.3f}" for a in accs))

C (weaker regularization = larger C)   clean   eps 0.05   eps 0.10   eps 0.20
C = 10                                 0.932    0.779    0.523    0.038
C = 1                                  0.915    0.803    0.614    0.091
C = 0.1                                0.897    0.811    0.676    0.223
C = 0.01                               0.873    0.799    0.666    0.336

Perturbations of 0.1 per pixel (on a 0 to 1 scale), visible as faint noise, cut the least regularized model's accuracy from 93 % to 52 %, and 0.2 reduces every model to near or below chance. Stronger regularization, which keeps weights small, trades a few points of clean accuracy for noticeably better accuracy under attack, an instance of the general tension between accuracy and robustness. For deep networks, iterative attacks such as projected gradient descent (PGD), which take many small steps within the allowed perturbation, are much stronger than FGSM, and robustness claims should be tested against them and against adaptive attacks designed for the specific defence.

Threat models

Robustness is always relative to a threat model: what the attacker knows and can change. White-box attackers know the model's parameters and can compute gradients; black-box attackers can only query the model, but can often transfer attacks crafted on a similar model or estimate gradients from queries. The perturbation set defines what changes are allowed: small changes in pixel values (measured by norms such as L∞ or L2), physical changes such as stickers on road signs, word substitutions in text, or arbitrary prompts to a language model. A defence that works against small pixel changes may be useless against other attacks. Clear threat models prevent false confidence.

Defences

Many proposed defences have been broken by stronger or adaptive attacks, which is a lesson in itself: robustness must be evaluated adversarially, not assumed. Defences that have held up include:

• Adversarial training: training on adversarial examples generated against the current model during training, so the model learns to resist them. PGD-based adversarial training, popularized by Madry and colleagues (2018), remains one of the most reliable defences, at the cost of extra training computation and some clean accuracy.
• Certified robustness: methods such as randomized smoothing or interval bound propagation that prove the prediction cannot change for any perturbation within a given size, providing guarantees rather than empirical claims, though usually for small perturbations.
• Input preprocessing and detection: transformations and detectors that remove or flag adversarial patterns, useful in combination but often bypassable alone.
• Ensembles and redundancy: requiring agreement from diverse models or sensors, so that one fooled component does not decide the outcome.
• System-level design: limiting what a model's output can do on its own, so that a fooled model cannot cause serious harm without other checks (Lesson 10).

Data poisoning and backdoors

Attackers can also act during training. In data poisoning, they insert crafted examples into the training data to degrade the model or cause specific errors. In a backdoor attack, the poisoned examples teach the model to behave normally except when a trigger, such as a particular pattern or phrase, appears, at which point it produces the attacker's chosen output. Because models are trained on huge scraped datasets, and fine-tuned on data from many sources, poisoning a small fraction of the data can be feasible, and studies have shown that even a small number of poisoned documents can implant backdoors in language models. Research on "sleeper agent" models has shown that some deliberately implanted backdoor behaviours can persist through standard safety training, which raises the stakes for data provenance, supply chain security (AI-420, Lesson 11) and detection methods, including interpretability (Lessons 6 and 7).

Robustness for language models

For language models, the input space is natural language, and attacks take the form of prompts: jailbreaks that elicit harmful content, prompt injections that hijack an agent's instructions, and inputs that trigger hallucination or inconsistent behaviour. Automated attacks, such as optimized adversarial suffixes found by gradient-guided search, showed that seemingly meaningless strings could reliably jailbreak aligned models and transfer between models. Defences combine training on adversarial prompts, input and output classifiers, and system design, and are evaluated through red teaming. These topics are the subject of Lesson 5.

Certified robustness and its limits

Empirical defences such as adversarial training are evaluated by attacking them, and history shows that many defences that looked strong were later broken by stronger, adaptive attacks designed specifically against them. Certified defences take a different approach: they provide a mathematical guarantee that no perturbation within a given budget can change the prediction for a particular input. Randomized smoothing, for example, classifies an input by the majority vote of the model on many noisy copies of it, and the margin of that vote yields a certified radius within which the prediction cannot change. Interval bound propagation and related methods compute bounds on a network's outputs over a whole region of inputs.

Certified methods give real guarantees, but only for the threat model they cover, usually small perturbations measured by a specific norm, and typically with lower clean accuracy and much higher computational cost. They do not address the threats that matter most for large language models, such as jailbreaks written in natural language, which are not small perturbations of anything. For those, robustness must be evaluated empirically and adaptively.

Evaluating robustness honestly

Good practice for robustness claims, developed after many broken defences, includes:

• Define the threat model precisely: what the attacker knows, what they can change, and their budget.
• Use adaptive attacks: design attacks that target the specific defence, rather than reusing attacks from earlier papers.
• Check for gradient masking: if black-box or random attacks succeed more often than gradient-based ones, the defence may be hiding gradients rather than being robust.
• Report accuracy as a function of attack strength, not at a single point, and include clean accuracy.
• Invite independent evaluation, since the authors of a defence are rarely its most creative attackers.

These principles carry over directly to evaluating safeguards for language models, where red teams play the role of adaptive attackers.

Robustness for language models

For large language models, robustness questions take new forms. Models may answer correctly when a question is phrased one way and incorrectly when it is paraphrased, translated into a less common language, or embedded in a long document. They may be sensitive to the order of multiple-choice options or to irrelevant details added to a problem. Evaluations therefore test consistency across paraphrases, languages, formats and distractors, and treat large performance drops under such changes as a warning that measured capability is brittle. These distribution shifts are often more important in practice than worst-case perturbations, because they occur naturally among ordinary users.