Privacy-preserving machine learning

Lesson 8 applied differential privacy to statistics. Machine learning raises the stakes: a model is a complex function of every training example, and modern models can memorize rare examples. This lesson covers the main technologies for training and using models while protecting the underlying data: differentially private stochastic gradient descent (DP-SGD), federated learning, secure aggregation and multiparty computation, homomorphic encryption, trusted execution environments, and synthetic data. These are often grouped as privacy-enhancing technologies (PETs). Each protects against different threats, and real systems often combine several.

Threat models first

Choosing a privacy technology starts with the question "private from whom?" Possible adversaries include an outsider who sees only the trained model or its API, a curious server operator who sees the training data or updates, other participants in a collaboration, and an attacker who compromises infrastructure. Differential privacy protects against what can be inferred from outputs, including the final model. Federated learning keeps raw data on devices but not, by itself, the information in the updates. Encryption and secure computation protect data during processing from the parties doing the computation. Matching technology to threat avoids both over-engineering and false confidence.

DP-SGD

Differentially private stochastic gradient descent, introduced by Abadi and colleagues in 2016, makes training differentially private with two changes to ordinary SGD:

• Per-example clipping: compute the gradient for each example in a batch and scale it down so its L2 norm is at most \(C\), bounding any one example's influence on the update.
• Noise addition: sum the clipped gradients and add Gaussian noise with standard deviation \(\sigma C\) before averaging and updating.

A privacy accountant tracks the cumulative \((\varepsilon, \delta)\) over all training steps, taking advantage of random sampling of batches, which amplifies privacy. The cost is accuracy: noise and clipping slow learning and reduce final accuracy, especially for small data sets and for under-represented groups, whose distinctive gradients are clipped and drowned by noise. Studies have found that DP-SGD can widen accuracy gaps between majority and minority groups, another place where privacy and fairness interact. Larger batches, pre-training on public data and fine-tuning privately, and careful tuning of \(C\) and \(\sigma\) narrow the gap. Libraries such as Opacus for PyTorch and TensorFlow Privacy implement DP-SGD with accountants.

Federated learning

Federated learning trains a shared model across many devices or organizations without collecting their raw data. In each round, a server sends the current model to a sample of clients; each client trains locally on its own data for a few steps and sends back the model update; the server averages the updates, weighted by data size, to produce the next model. This is the federated averaging (FedAvg) algorithm of McMahan and colleagues. It is used for keyboard prediction on phones, for hospitals collaborating on medical imaging models, and for banks detecting fraud across institutions.

Federated learning is not private by itself. Model updates can leak information about the local data; gradient inversion attacks have reconstructed images and text from shared gradients. Federated learning therefore usually combines with secure aggregation, so the server sees only the sum of updates, and with differential privacy, by clipping each client's update and adding noise to the aggregate, which gives client-level privacy: the model barely depends on whether any one client participated.

import numpy as np
# Federated averaging across 20 clinics, with and without clipped, noised updates
rng = np.random.default_rng(9)
d = 5
w_true = np.array([1.5, -2.0, 1.0, 0.5, -0.3])
clients = []
for c in range(20):
    m = rng.integers(100, 400)
    shift = rng.normal(0, 0.5, d)                         # each clinic sees a different patient mix
    x = rng.normal(0, 1, (m, d)) + shift
    y = (rng.random(m) < 1 / (1 + np.exp(-x @ w_true))) * 1.0
    clients.append((x, y))
x_test = rng.normal(0, 1, (5000, d)); y_test = (x_test @ w_true) > 0

def local_update(w, x, y, epochs=5, lr=0.1):
    w = w.copy()
    for _ in range(epochs):
        p = 1 / (1 + np.exp(-x @ w))
        w -= lr * x.T @ (p - y) / len(y)
    return w

def train(rounds, clip=None, sigma=0.0):
    w = np.zeros(d)
    for r in range(rounds):
        picked = rng.choice(20, 10, replace=False)
        deltas = []
        for c in picked:
            delta = local_update(w, *clients[c]) - w
            if clip is not None:
                delta *= min(1.0, clip / np.linalg.norm(delta))   # bound each clinic influence
            deltas.append(delta)
        avg = np.mean(deltas, axis=0)
        if sigma > 0:
            avg += rng.normal(0, sigma * clip / len(picked), d)  # Gaussian noise for client level DP
        w += avg
    return w

def acc(w):
    return np.mean((x_test @ w > 0) == y_test)

print("setting                           test accuracy  weight error")
for label, kw in [("federated averaging", {}), ("clipped, no noise", {"clip": 0.5}),
                  ("clipped + noise sigma 0.5", {"clip": 0.5, "sigma": 0.5}),
                  ("clipped + noise sigma 2.0", {"clip": 0.5, "sigma": 2.0})]:
    w = train(100, **kw)
    err = np.linalg.norm(w / np.linalg.norm(w) - w_true / np.linalg.norm(w_true))
    print(f"{label:32s}  {acc(w):13.3f}  {err:12.3f}")
print("raw patient records never leave the clinics; only model updates are shared")

setting                           test accuracy  weight error
federated averaging                       0.985         0.044
clipped, no noise                         0.985         0.041
clipped + noise sigma 0.5                 0.980         0.048
clipped + noise sigma 2.0                 0.938         0.175
raw patient records never leave the clinics; only model updates are shared

In the simulation, each clinic has a different patient mix, a non-identical data distribution that is the normal case in federated settings. Plain federated averaging reaches 98.5 % test accuracy without any clinic sharing records. Clipping updates barely changes accuracy. Adding noise with multiplier 0.5 costs half a percentage point; multiplier 2.0, which gives a much stronger privacy guarantee, drops accuracy to 93.8 % and moves the learned weights noticeably. With only 10 clinics per round, the noise is large relative to the signal; real deployments with thousands of participating devices per round can use strong noise with little loss, which is why client-level DP works well at the scale of phone keyboards but is harder for a few hospitals.

Challenges in federated learning

Beyond privacy, federated learning faces practical difficulties. Client data are non-identically distributed, which slows convergence and can produce a global model that fits some clients poorly; personalization techniques fine-tune the global model locally. Devices drop out, have limited bandwidth and battery, and vary in hardware. Malicious clients can send poisoned updates, so robust aggregation methods such as trimmed means or median aggregation are used. Fairness across clients matters: the global model may serve large, well-represented clients best. And evaluation is hard because the server cannot see the test data.

Cryptographic approaches

Cryptography protects data while it is being computed on:

• Secure multiparty computation (MPC) lets several parties compute a function of their combined inputs while revealing only the output, for example two banks computing overlap in suspected fraud accounts without disclosing their customer lists. Secret sharing splits each value into random shares held by different parties.
• Secure aggregation is a specialized, efficient MPC protocol that lets a server learn only the sum of client updates.
• Homomorphic encryption allows computation on encrypted data: a client encrypts its input, the server runs a model on the ciphertext, and only the client can decrypt the result. Fully homomorphic encryption supports arbitrary computation but is still orders of magnitude slower than plaintext, so it is used for small models or specific operations.
• Private set intersection lets two parties find common records without revealing the rest.

Cryptographic methods protect inputs during computation but not what the output reveals; a model trained with MPC can still memorize data, so they are often combined with differential privacy.

Trusted execution environments

Trusted execution environments (TEEs), such as Intel SGX and TDX, AMD SEV and confidential computing on recent GPUs, run code in hardware-isolated enclaves whose memory is encrypted and whose code can be verified by remote attestation. Data owners can confirm exactly which code will process their data before sending it. TEEs are much faster than homomorphic encryption and are used for confidential inference and training in the cloud, but they rely on trust in the hardware vendor and have been affected by side-channel attacks.

Synthetic data

Synthetic data generators learn the distribution of real data and produce artificial records for sharing or development. Synthetic data are not automatically private: a generator can memorize and reproduce real records, especially outliers. Generators trained with differential privacy, such as DP versions of GANs or of marginal-based methods, provide formal guarantees, at a cost in fidelity for complex relationships and small subgroups. Synthetic data are most useful for software testing, prototyping and education, and less reliable for final statistical conclusions, which should be validated on real data.

Combining technologies in practice

Production systems rarely rely on a single PET. A typical federated keyboard model combines on-device training, secure aggregation so the server never sees individual updates, client-level differential privacy with a published epsilon, and filtering so rare words that could identify someone are not learned. A data collaboration between companies might use a trusted execution environment to run an agreed analysis on pooled data, attestation so each party can verify the code, and differential privacy on the outputs before they are shared. A medical research network might combine federated analysis for routine statistics with a secure data enclave for approved researchers. Each layer addresses a different part of the threat model, and each has its own failure modes, so the design should document which layer protects against which threat.

Governance matters as much as technology. Agreements must specify what analyses are permitted, who approves new queries, how the privacy budget is allocated among partners, and what happens when the budget is exhausted. Without these rules, a technically strong system can still be used in ways that defeat its purpose, for example by running many slightly different analyses until a sensitive detail emerges.

Evaluating privacy technologies

Claims about PETs should be tested. For DP systems, check the accounting and run membership inference attacks (Lesson 10) to confirm that empirical leakage is within the bound. For federated systems, test gradient inversion attacks against the shared updates with and without secure aggregation and noise. For synthetic data, measure how close generated records are to real ones and whether rare records reappear. Publishing these evaluations, as some organizations now do in privacy reports, lets outsiders assess whether the protection is real.

Finally, privacy technologies have operational costs that should be weighed early. Federated systems need client software, update scheduling and monitoring across devices; secure computation adds communication rounds that can dominate training time; homomorphic encryption may make inference hundreds of times slower; and DP training typically needs larger batches and more tuning. Budgeting for these costs at the design stage avoids the common outcome in which a privacy technology is promised in a proposal but quietly dropped when it proves expensive in production.