Data governance, privacy and security

Data used for AI is often about people: customers, patients, employees, students, drivers. Using it well means knowing who is allowed to do what with which data, for which purpose and for how long; protecting it from people who should not see it; and reducing what can be learned about any individual from the data sets, analyses and models built from it. These concerns are not obstacles bolted onto the technical work. They shape what data can be collected (Lesson 1), how it is labelled and shared (Lesson 6), how pipelines store it (Lesson 8) and what can be published. This lesson covers data governance, the main privacy laws practitioners meet in Canada and the United States, de-identification and its limits, differential privacy, and the security controls that protect data in use.

Data governance

Data governance is the set of roles, policies and processes that decide how an organization's data is managed. Its practical elements are:

• Ownership and stewardship: each data set has an accountable owner (who decides who may use it) and stewards (who maintain its quality and documentation).
• Classification: data is labelled by sensitivity, for example public, internal, confidential and restricted, with personal and health information in the highest classes. The class decides where data may be stored, who may access it and how it must be protected.
• A data catalogue: a searchable inventory of data sets with their owners, descriptions, classifications, lineage and terms of use, so people can find data and know whether they may use it.
• Purpose and consent: personal data is collected for stated purposes, and using it for a new purpose, such as training a model, may require new consent or a legal basis.
• Retention and deletion: data is kept only as long as needed and then deleted, including copies in backups, extracts and feature stores.
• Quality and lineage: knowing where data came from and how it was transformed (Lesson 8).

Governance is what lets an AI team answer the questions that regulators, auditors and customers ask: where did this training data come from, were people told, who can see it, and can you delete a person's data when they ask?

Privacy law in brief

Laws differ by jurisdiction and sector, and change often, so practitioners work with legal and privacy advisers. The main frameworks a North American AI practitioner meets are summarized below (as of 2026):

Common threads run through all of them: collect only what is needed, use it for the stated purpose, protect it, let people see and correct it, delete it when it is no longer needed, and report breaches. For AI work, the questions that come up most often are whether training a model is a compatible purpose, whether a model or its outputs contain personal information, and how to honour deletion requests for data already used in training.

De-identification and its limits

Removing names and account numbers does not make data anonymous. Combinations of ordinary attributes, called quasi-identifiers, such as postal area, birth date and sex, are often unique to one person and can be matched against other data sets that do include names. This is a linkage attack.

import pandas as pd

# A "de-identified" health extract: names removed, quasi-identifiers kept
health = pd.DataFrame({
    "postal3": ["T2N", "T2N", "M5V", "M5V", "H2X"],
    "birth_year": [1984, 1991, 1984, 1975, 1991],
    "sex": ["F", "M", "F", "F", "M"],
    "diagnosis": ["asthma", "diabetes", "depression", "asthma", "migraine"],
})
# A public or purchased list with names and the same attributes
voters = pd.DataFrame({
    "name": ["R. Singh", "J. Tremblay", "K. Okafor", "L. Chen", "M. Roy", "P. Novak",
             "A. Martin", "S. Gill", "D. Lee", "E. Roy", "F. Wong"],
    "postal3": ["T2N", "T2N", "M5V", "M5V", "H2X", "M5V", "T2N", "T2N", "M5V", "H2X", "M5V"],
    "birth_year": [1984, 1991, 1984, 1975, 1991, 1975, 1988, 1993, 1986, 1999, 1979],
    "sex": ["F", "M", "F", "F", "M", "F", "F", "M", "F", "M", "F"],
})

keys = ["postal3", "birth_year", "sex"]
k = health[keys].merge(voters.groupby(keys).size().rename("k").reset_index(), on=keys, how="left")["k"]
print("matching people per record (exact year):", k.tolist())
linked = health[k.to_numpy() == 1].merge(voters, on=keys)
print(linked[["name", "diagnosis"]].to_string(index=False))

# Generalize birth year to a decade and measure again
for df in (health, voters):
    df["decade"] = df["birth_year"] // 10 * 10
keys2 = ["postal3", "decade", "sex"]
k2 = health[keys2].merge(voters.groupby(keys2).size().rename("k").reset_index(), on=keys2, how="left")["k"]
print("matching people per record (decade):    ", k2.tolist())
print("smallest group size k =", int(k2.min()))

matching people per record (exact year): [1, 1, 1, 2, 1]
       name  diagnosis
   R. Singh     asthma
J. Tremblay   diabetes
  K. Okafor depression
     M. Roy   migraine
matching people per record (decade):     [2, 2, 2, 3, 2]
smallest group size k = 2

Generalizing quasi-identifiers until every combination is shared by at least k people gives k-anonymity (Lesson 6). It reduces the risk of singling someone out, but has known weaknesses: if everyone in a group has the same diagnosis, the diagnosis is still revealed (addressed by l-diversity), and an attacker with extra knowledge can still narrow things down. Real de-identification combines several techniques:

• Removing direct identifiers (names, numbers, emails, exact addresses, face images).
• Generalizing quasi-identifiers (dates to months or years, locations to larger areas, ages to bands).
• Suppressing rare records or values that remain unique.
• Masking identifiers in free text and images.
• Controlling access: sharing de-identified data only under agreements that prohibit re-identification, in secure environments.

Pseudonymization

Pseudonymization replaces identifiers with consistent tokens, so records about the same person can still be linked across tables, without revealing who they are. It is not the same as anonymization: whoever holds the mapping can reverse it, and the data remains personal information under most laws. A plain hash is a poor pseudonym, because anyone can hash a list of likely values and compare.

import hashlib
import hmac
import secrets

emails = ["amira@example.com", "ben@example.org", "amira@example.com"]

# A plain hash is NOT anonymous: anyone can hash a list of known emails and match
plain = [hashlib.sha256(e.encode()).hexdigest()[:16] for e in emails]
guess = hashlib.sha256("amira@example.com".encode()).hexdigest()[:16]
print("plain hashes:", plain)
print("attacker matches amira by hashing a guess:", guess == plain[0])

# A keyed hash (HMAC) with a secret key kept separately cannot be recomputed without the key
key = secrets.token_bytes(32)
def pseudonym(value):
    return hmac.new(key, value.encode(), hashlib.sha256).hexdigest()[:16]
tokens = [pseudonym(e) for e in emails]
print("keyed tokens are consistent for joins:", tokens[0] == tokens[2])
print("attacker without the key cannot match:", hashlib.sha256("amira@example.com".encode()).hexdigest()[:16] in tokens)

plain hashes: ['0068ffbd2f832e74', '9de4dcc67240d506', '0068ffbd2f832e74']
attacker matches amira by hashing a guess: True
keyed tokens are consistent for joins: True
attacker without the key cannot match: False

The key must be stored separately from the data, with strict access, and rotated according to policy. If the same key is used across projects, tokens can be joined across them, which may or may not be intended.

Differential privacy

Differential privacy (DP) offers a mathematical guarantee rather than a set of heuristics. A randomized algorithm is ε-differentially private if its output distribution changes by at most a factor of \(e^{\varepsilon}\) when any one person's data is added or removed. Whatever an attacker knows, the output reveals little about whether any individual was included. The parameter ε (epsilon) is the privacy loss: smaller is more private but noisier.

The simplest DP algorithm is the Laplace mechanism for counts: add random noise drawn from a Laplace distribution with scale \(\Delta / \varepsilon\), where \(\Delta\) is the sensitivity, the most one person can change the answer (1 for a count).

import numpy as np

rng = np.random.default_rng(42)
true_count = 37        # patients with a condition in one small area

# Laplace mechanism for a count: one person changes it by at most 1 (sensitivity 1)
def dp_count(count, epsilon, n=1):
    return count + rng.laplace(0.0, 1.0 / epsilon, n)

for eps in (0.1, 0.5, 1.0, 2.0):
    draws = dp_count(true_count, eps, 10000)
    mae = np.mean(np.abs(draws - true_count))
    print(f"epsilon {eps:4.1f}: typical error {mae:5.1f}, one release = {dp_count(true_count, eps)[0]:6.1f}")

# Privacy budget: answering k queries at epsilon each spends k * epsilon in total
budget, per_query = 1.0, 0.25
print("queries allowed within the budget:", int(budget / per_query))

epsilon  0.1: typical error  10.0, one release =   42.8
epsilon  0.5: typical error   2.0, one release =   46.6
epsilon  1.0: typical error   1.0, one release =   40.5
epsilon  2.0: typical error   0.5, one release =   35.0
queries allowed within the budget: 4

Two properties make DP practical. Privacy loss adds up across queries (composition), so an organization sets a total budget and tracks spending. And any processing of a DP output stays DP (post-processing), so released statistics can be rounded, summed or plotted freely. DP is used by national statistical agencies for census data, by technology companies for usage statistics, and in machine learning through DP-SGD, which clips and adds noise to gradients during training so that the model itself is differentially private. The cost is accuracy, which is acceptable for large populations and aggregate statistics, and difficult for small groups.

Security of data in use

Privacy rules mean little if data leaks. The main security controls for data science environments are:

• Least privilege: people and services get access only to the data they need, preferably to views or extracts rather than raw tables, and access is reviewed and removed when no longer needed.
• Encryption at rest (on disks and in object storage) and in transit (TLS for every connection), with keys managed in a key-management service.
• Secure environments: sensitive data is analysed in controlled workspaces (sometimes called data clean rooms or trusted research environments) rather than downloaded to laptops; outputs are checked before they leave.
• Logging and monitoring of access to sensitive data, so misuse can be detected and investigated.
• Secrets management: database passwords, API keys and hashing keys stored in a vault, never in notebooks, code repositories or configuration files committed to version control.
• Minimization in pipelines: removing or tokenizing sensitive fields as early as possible, so downstream systems never hold them.

Models can leak data

Trained models can memorize and reveal training data. Membership inference attacks test whether a particular record was in the training set; large language models have been shown to reproduce rare strings, such as phone numbers, from their training data. Defences include de-duplicating and filtering training data, removing personal information before training, regularization, DP training for sensitive data, and testing models for memorization before release. Courses AI-410 and AI-460 return to these risks.

Honouring requests and deleting data

People may ask to see, correct or delete their data. Meeting such a request requires knowing where every copy is: source systems, warehouses, extracts, feature stores, labelled data sets, backups and logs. Lineage and the catalogue make this possible. For models already trained, organizations document how deletion is handled: retraining on schedule without the deleted data, or, for some regulated uses, retraining promptly. Designing pipelines so that training sets are rebuilt from governed sources, rather than from untracked extracts, is what makes deletion achievable.