Privacy and data protection for AI

Privacy law is the body of law most often applied to AI today. AI systems consume personal information in training, process it when making predictions, generate new information about people through inferences, and can sometimes reproduce personal data they were trained on. Privacy regulators have been among the most active enforcers in the AI field, investigating facial recognition, generative AI services and data scraping. This lesson explains how privacy principles apply across the AI life cycle, the legal frameworks in Canada, the United States and Europe, the special issues of scraped and publicly available data, de-identification and its limits, individual rights and the challenge of deleting data from trained models, privacy impact assessments, and practical privacy engineering.

Personal information across the AI life cycle

Personal information is information about an identifiable individual. It appears at every stage of an AI system:

• Training data: customer records, transcripts, images, text scraped from the web, medical records, location traces.
• Inputs at inference: the data a user or system provides to get a prediction or response, including prompts that contain personal details.
• Outputs and inferences: predictions about a person (credit risk, health risk, interests, personality), which are themselves personal information, sometimes sensitive, even if derived from innocuous inputs.
• Model parameters: large models can memorize and reproduce fragments of training data, including names, contact details or rare personal facts, and attacks can sometimes extract them (AI-430).
• Logs and feedback: conversation histories and user feedback retained for improvement.

Each stage raises questions of legal authority (consent or another basis), purpose, minimization, retention, security and individual rights.

Privacy principles applied to AI

Privacy laws around the world share principles derived from the OECD privacy guidelines and fair information practices:

• Accountability: an identified person or function is responsible for compliance.
• Identifying purposes and purpose limitation: data collected for one purpose (providing a service) cannot simply be used for another (training a model for a different product) without consent or another legal basis.
• Consent or lawful basis: meaningful consent, appropriate to the sensitivity of the data and the reasonable expectations of the individual, or another basis permitted by law.
• Limiting collection (minimization): collect only what is needed for the identified purpose.
• Limiting use, disclosure and retention: keep data only as long as needed.
• Accuracy: data used to make decisions about people should be accurate and up to date.
• Safeguards: security appropriate to sensitivity.
• Openness: clear information about practices, including the use of AI.
• Individual access and correction: people can see and correct information about them.
• Challenging compliance: a route to complain.

AI strains several of these: large models seem to require large amounts of data, purposes for training are often broad and evolving, and inferences can reveal sensitive information people never disclosed.

Legal frameworks

In Canada, the Personal Information Protection and Electronic Documents Act (PIPEDA) governs private-sector organizations in commercial activities, except in provinces with substantially similar laws (Quebec, Alberta and British Columbia), which apply within those provinces; public bodies are covered by federal and provincial public-sector privacy acts, and health information by provincial health privacy laws. PIPEDA is based on consent and on the reasonable person test: organizations may collect, use and disclose personal information only for purposes a reasonable person would consider appropriate in the circumstances. The Office of the Privacy Commissioner (OPC) investigates complaints but under PIPEDA cannot issue fines. Quebec's Law 25 modernized its private-sector law with stronger consent, mandatory privacy impact assessments for certain projects and for transfers outside Quebec, rules on anonymization, transparency for automated decisions (Lesson 2), and administrative monetary penalties up to 10 million dollars or 2 % of worldwide turnover (penal fines up to 25 million dollars or 4 %). Bill C-36, the Protecting Privacy and Consumer Data Act, introduced in June 2026, would modernize PIPEDA: recognizing privacy as a fundamental right, strengthening consent, adding deletion rights (including for AI deepfakes), protecting children's information as sensitive, requiring privacy impact assessments, requiring explanations of automated decision-making, and creating a Digital Safety and Data Protection Commission with penalties up to 10 million dollars or 3 % of global revenue, and up to 25 million dollars or 5 % for the most serious offences. The OPC and provincial commissioners issued joint principles for responsible generative AI in 2023.

The EU General Data Protection Regulation (GDPR) applies to organizations anywhere that offer goods or services to people in the EU or monitor their behaviour. It requires a lawful basis for processing (consent, contract, legitimate interests and others), imposes strict rules on special categories of data, gives individuals rights including a right not to be subject to decisions based solely on automated processing with legal or similarly significant effects (Article 22), and carries fines up to 4 % of worldwide turnover. European regulators have issued opinions on training AI models with personal data, generally allowing legitimate interests as a basis where balancing tests and safeguards are met.

Scraped and publicly available data

Many AI models are trained on data scraped from the web, including personal information in social media posts, photos and forums. Publicly accessible does not mean free of privacy obligations. Privacy regulators from many countries issued a joint statement in 2023 that scraping personal information can breach privacy laws and that platforms must protect users against unlawful scraping. Canadian commissioners found in 2021 that Clearview AI violated privacy laws by scraping billions of facial images to build a recognition tool without consent, and other regulators reached similar conclusions; the company faced fines in Europe and settlements in the United States. Developers training on web data should assess the legal basis in each jurisdiction, respect technical opt-outs, filter sensitive categories and personal identifiers where possible, and be prepared to honour individual rights.

De-identification and its limits

Removing names and direct identifiers does not make data anonymous. Combinations of quasi-identifiers, such as age, location and sex, can single out individuals, and linkage with other datasets can re-identify them. Privacy laws distinguish between de-identified data (identifiers removed but re-identification possible, still personal information in most laws) and anonymized data (irreversibly processed so that individuals cannot be identified, generally outside privacy law), and set standards for each.

import numpy as np
# Re-identification risk in a "de-identified" training dataset: how many records are unique on quasi-identifiers?
rng = np.random.default_rng(4)
n = 50_000
age = rng.integers(18, 90, n)
postal3 = rng.integers(0, 400, n)            # first three characters of a postal or ZIP code, 400 areas
sex = rng.integers(0, 2, n)
def k_report(cols, label):
    keys = np.stack(cols, axis=1)
    _, inv, counts = np.unique(keys, axis=0, return_inverse=True, return_counts=True)
    k = counts[inv.ravel()]
    print(f"{label:38s} unique {np.mean(k == 1):6.1%}   k below 5 {np.mean(k < 5):6.1%}")
k_report([age, postal3, sex], "exact age, area, sex")
k_report([age // 10, postal3, sex], "age in 10-year bands, area, sex")
k_report([age // 10, postal3 // 10, sex], "age bands, 40 regions, sex")

exact age, area, sex                   unique  42.1%   k below 5  99.0%
age in 10-year bands, area, sex        unique   0.5%   k below 5   5.2%
age bands, 40 regions, sex             unique   0.0%   k below 5   0.0%

With exact age, a 400-area location code and sex, 42 % of records are unique and almost all are in groups smaller than five, making re-identification easy for anyone who knows those three facts about a person. Generalizing age to ten-year bands cuts uniqueness to under 1 %, and generalizing location to 40 regions eliminates small groups entirely. This is the logic of k-anonymity: every record should be indistinguishable from at least k−1 others on its quasi-identifiers. k-anonymity is not a complete protection, since attributes within a group may still reveal sensitive information, so other techniques complement it: aggregation, noise addition, differential privacy (mathematical limits on what any output reveals about one individual), and synthetic data generated to preserve statistical properties without real records. Each reduces data utility, so the right choice depends on the use.

Individual rights and machine unlearning

Individuals have rights to access, correct and, increasingly, delete their personal information. For AI, these rights raise hard technical questions. Deleting a record from a database is straightforward; removing its influence from a trained model is not. Options include retraining without the data (expensive for large models), techniques for machine unlearning (an active research area with limited guarantees), filtering outputs so that the model does not reproduce the person's information, and designing systems so that personal data stays in retrievable stores (such as retrieval databases) rather than in model weights, where it can be deleted. Regulators have begun to address these questions, and organizations should document how they handle rights requests for AI systems.

Privacy impact assessments

A privacy impact assessment (PIA), or data protection impact assessment, systematically identifies the personal information involved in a project, its flows, the legal authority, the risks to individuals, and mitigations, before the project proceeds. PIAs are mandatory in many circumstances (for example under GDPR for high-risk processing, under Quebec's Law 25 for many projects, under state privacy laws for profiling, and in the public sector) and are good practice everywhere. For AI projects, the PIA should cover training data sources and rights, inference data, inferences generated, retention, vendor access, cross-border transfers, and how rights requests and model updates will be handled. PIAs often combine with algorithmic impact assessments (Lesson 2).

Privacy engineering for AI

Practical measures that build privacy into AI systems include:

• Minimize: use only the fields needed; prefer aggregated or de-identified data for training.
• Separate: keep identifiers apart from training data, with controlled linkage.
• Filter: remove direct identifiers and sensitive data from training corpora and from logs; detect and redact personal information in prompts and outputs.
• Protect: encryption, access controls and audit logs; contractual limits on vendors' use of data, including commitments not to train on customer data.
• Limit retention: delete inference logs and conversation histories after defined periods unless needed.
• Use privacy-enhancing technologies: differential privacy in training, federated learning, secure enclaves.
• Test: probe models for memorization and extraction of personal data before release.