Privacy attacks on models and machine unlearning

A trained model is a compressed summary of its training data, and summaries can leak. Over the past decade researchers have shown that attackers can tell whether a person was in a training set, reconstruct representative inputs, recover sensitive attributes, and extract verbatim training examples, including names, phone numbers and copyrighted text, from large language models. These attacks matter for compliance, because laws treat leakage of personal data from models seriously, and they guide defence, because the same attacks are the standard way to audit a model's privacy. This lesson explains the main attacks, demonstrates membership inference in code, and discusses defences and machine unlearning, the problem of removing a person's data from a trained model.

Membership inference

A membership inference attack decides whether a given record was in the model's training set. Knowing that someone's record was used to train a model for a particular disease, for example, can reveal that they have the disease. The attack exploits the fact that models usually behave differently on training data than on unseen data: they are more confident and have lower loss on examples they were trained on. Shokri and colleagues introduced the attack in 2017 using shadow models, models the attacker trains on similar data to learn what member and non-member outputs look like. Simpler attacks threshold the loss or confidence directly, and state-of-the-art attacks such as LiRA (the likelihood ratio attack) calibrate the threshold for each example using many shadow models, which detects members far more reliably at low false positive rates.

import numpy as np
# Loss threshold membership inference: overfitting leaks who was in the training set
rng = np.random.default_rng(10)
d = 100
w_true = rng.normal(0, 1, d) * (rng.random(d) < 0.2)
def make(m):
    x = rng.normal(0, 1, (m, d))
    y = (rng.random(m) < 1 / (1 + np.exp(-x @ w_true))) * 1.0
    return x, y

def fit(x, y, lam, steps=4000, lr=0.5):
    w = np.zeros(d)
    for _ in range(steps):
        p = 1 / (1 + np.exp(-x @ w))
        w -= lr * (x.T @ (p - y) / len(y) + lam * w)
    return w

def losses(w, x, y):
    p = np.clip(1 / (1 + np.exp(-x @ w)), 1e-9, 1 - 1e-9)
    return -(y * np.log(p) + (1 - y) * np.log(1 - p))

print("model                      train_acc  test_acc  attack_acc  advantage")
for label, n, lam in (("no regularization, n=150", 150, 0.0), ("L2 regularized, n=150", 150, 0.6),
                      ("L2 regularized, n=3000", 3000, 0.6)):
    x_in, y_in = make(n)          # members (training set)
    x_out, y_out = make(n)        # non-members from the same population
    w = fit(x_in, y_in, lam)
    li, lo = losses(w, x_in, y_in), losses(w, x_out, y_out)
    tr = np.mean(((x_in @ w) > 0) == y_in); te = np.mean(((x_out @ w) > 0) == y_out)
    # the attacker guesses "member" when the loss is below a threshold; report the best threshold
    cands = np.quantile(np.r_[li, lo], np.linspace(0.01, 0.99, 99))
    best = max(0.5 * (np.mean(li < t) + np.mean(lo >= t)) for t in cands)
    print(f"{label:25s}  {tr:9.2f}  {te:8.2f}  {best:10.2f}  {2 * best - 1:9.2f}")
print("advantage 0 means the attacker does no better than a coin flip")

model                      train_acc  test_acc  attack_acc  advantage
no regularization, n=150        1.00      0.76        0.68       0.36
L2 regularized, n=150           0.89      0.67        0.65       0.29
L2 regularized, n=3000          0.88      0.87        0.52       0.03
advantage 0 means the attacker does no better than a coin flip

The unregularized model trained on 150 records with 100 features fits its training data perfectly (100 % training accuracy against 76 % on new data). The attacker, guessing "member" whenever the loss is below a threshold, is right 68 % of the time, an advantage of 0.36 over coin flipping. L2 regularization reduces the advantage only to 0.29, because with so few records the model still fits them much better than new ones. Training on 3,000 records nearly closes the gap between training and test accuracy, and the attack advantage falls to 0.03. The lesson generalizes: membership inference thrives on overfitting, and the strongest defences are more data, regularization and, for a formal guarantee, differential privacy, since \(\varepsilon\)-DP guarantees that any attack's true positive rate is at most \(e^{\varepsilon}\) times its false positive rate (plus \(\delta\) in the approximate version).

Evaluating attacks properly

Average attack accuracy can understate risk. An attack that is only slightly better than chance on average might identify a few members with near certainty, and those few people are exposed. Carlini and colleagues therefore recommend reporting the true positive rate at a very low false positive rate, such as 0.1 %, which shows whether any members can be confidently identified. Outliers and rare examples are most vulnerable, because the model must memorize them to fit them. Privacy auditing tools such as ML Privacy Meter and the auditing features of some DP libraries run these attacks systematically.

Attribute inference and model inversion

Attribute inference uses a model to infer a sensitive attribute of a person whose other attributes are known. Fredrikson and colleagues showed that a model predicting warfarin dosage from genetic markers and demographics could be used, with a patient's demographics and dose, to infer their genetic markers better than chance. Model inversion reconstructs representative inputs for a class: from a face recognition model that outputs confidence scores, attackers reconstructed recognizable images of people in the training set. Newer gradient inversion attacks in federated learning (Lesson 9) reconstruct actual training images and text from shared gradients. Limiting outputs to labels rather than confidence scores, and adding differential privacy, reduce these risks.

Training data extraction from language models

Large language models memorize some of their training data verbatim, particularly text that appears many times or is distinctive. Carlini and colleagues extracted hundreds of memorized sequences from GPT-2, including names, phone numbers, email addresses and code, by generating many samples and ranking them by how much more confident the model was than a reference model. Later work extracted gigabytes of training data from production models using a "divergence" prompt that caused the model to stop behaving like a chatbot and emit memorized text. Memorization grows with model size, with the number of times an example is duplicated in training data, and with the length of the prompt context given.

Defences include deduplicating training data, which greatly reduces memorization; filtering personal information and secrets from corpora; output filters that block known personal data patterns; differential privacy in fine-tuning on sensitive data; and testing models with extraction attacks before release. Organizations fine-tuning models on customer support logs, medical notes or code repositories should assume that rare strings in that data, such as account numbers or keys, could be reproduced unless they take these measures.

Model extraction

Model extraction, or model stealing, is an attack on the model itself rather than its training data: an attacker queries a model through its API and trains a copy that imitates it. Extraction threatens intellectual property, and the copy can then be used to craft adversarial examples or to run privacy attacks offline. Defences include rate limiting, monitoring for unusual query patterns, returning less information (labels instead of probabilities), and watermarking outputs. Extraction overlaps with security topics covered in AI-440 and AI-430.

Machine unlearning

Data protection laws give people the right to have their personal data erased in many circumstances. For a database, erasure means deleting a row. For a trained model, the data have been blended into the weights, so deleting the original record does not remove its influence. Machine unlearning aims to produce a model that behaves as though a person's data had never been used. Approaches include:

• Exact retraining: retrain from scratch without the data. This is the gold standard but very expensive for large models.
• Sharded training (SISA): split the training data into shards, train a separate model on each and combine them; to forget a record, retrain only the shard that contained it.
• Approximate unlearning: adjust the weights to remove a record's influence, for example with gradient ascent on the forgotten data or influence function updates, followed by verification.
• Differential privacy as a buffer: a model trained with DP is, by definition, barely influenced by any one record, which limits the harm of not unlearning.

Verifying unlearning is difficult. A common check is to run membership inference on the forgotten data: after unlearning, those records should look like non-members. For large language models, unlearning specific knowledge, such as a person's details or hazardous information, is an active research area, and current methods often suppress outputs rather than truly removing information.

In Canada, Quebec's Law 25 grants rights to have personal information de-indexed and to have it destroyed when its purpose is fulfilled, and federal privacy reform proposals have included a right to request disposal; regulators have also signalled that organizations must be able to honour deletion requests across systems that use the data.

Building a privacy evaluation into the pipeline

A practical privacy evaluation for a model before release includes: measuring the generalization gap; running loss-based and calibrated membership inference attacks, reporting true positive rates at low false positive rates; testing for verbatim extraction of known sensitive strings (canaries can be inserted deliberately into training data to measure memorization); checking what the API returns; and documenting results in the model card. If a model will be fine-tuned on sensitive data, compare these measures with and without DP to make an informed choice.

Privacy auditing with canaries

A practical way to measure memorization is to insert canaries, artificial secrets such as random sequences of digits formatted like phone numbers, into the training data a known number of times, train the model, and then test how readily the model reproduces them. Carlini and colleagues defined an exposure metric that compares the model's likelihood of the inserted canary with its likelihood of many random alternatives. High exposure means the model has memorized the canary and would likely memorize real secrets that appear similarly often. Canary tests are cheap, work for any model that assigns likelihoods, and give a concrete number to track across training runs and mitigation choices. Recent work has also used canaries to audit differentially private training empirically, checking that the measured leakage is consistent with the claimed epsilon, which has exposed bugs in some implementations.

Privacy risk in deployed applications

Many privacy failures in applications built on language models do not involve training at all. A support assistant with retrieval access to a customer database can be manipulated by prompt injection into revealing another customer's records; conversation logs stored for quality review can expose sensitive disclosures; and a model given broad tool access can send data to places it should not. Access controls that limit retrieval to the current user's data, minimal logging with retention limits, and testing with adversarial prompts are as important as defences against training data extraction.