Privacy and data

Brightline Analytics has asked Riverton General for more data. To improve Pathfinder, the company wants five years of emergency records, including diagnoses, test results, ambulance notes and the postal codes of patients' homes. It promises to remove names and health card numbers. The hospital's privacy officer is uneasy, and Varga brings the request to the seminar. 'No names,' Theo says. 'What's the problem?' Raúl, who has worked in emergency for eleven years, says he could probably identify half the patients in that data set from the ambulance notes alone. Sabine asks a different question: even if no one is identified, did patients who came to the emergency department expect their records to train a company's product?

Privacy is one of the oldest and most contested values in technology ethics. This lesson asks what privacy is, why it matters, why removing names is often not enough, and how the law protects it. The next lesson turns to surveillance, where privacy meets power.

What is privacy?

There is no single agreed definition of privacy, and philosophers have offered several. Each captures part of what people care about.

• Privacy as being let alone. In 1890 Samuel Warren and Louis Brandeis, writing in the Harvard Law Review after the spread of portable cameras and popular newspapers, described a right to be let alone and argued that the law should protect people from unwanted publicity about their private lives.
• Privacy as control. Alan Westin, writing in 1967, defined privacy as the claim of individuals to determine for themselves when, how and to what extent information about them is communicated to others. This view underlies consent-based data protection laws.
• Privacy as limited access. Some philosophers define privacy as the condition of being inaccessible to others in certain ways, whether or not you control it; a person stranded on an island has privacy without exercising control.
• Privacy as contextual integrity. Helen Nissenbaum argued that information flows carry norms tied to the context in which they occur, and that privacy is violated when a flow breaks those norms.

Contextual integrity is especially useful for data and AI, so the seminar spends time on it. Nissenbaum's insight is that the same piece of information can be shared appropriately in one context and inappropriately in another. A patient expects that what she tells an emergency nurse will be shared with the doctor treating her, recorded in her chart and perhaps used to audit the quality of care. She may not expect it to be sold to an insurer, posted publicly or used to train a commercial product. The information has not changed; the flow has. To evaluate a flow, contextual integrity asks who is sending the information, who is receiving it, whom it is about, what kind of information it is and under what principle it is transmitted, such as confidentiality, consent, legal requirement or sale.

Why privacy matters

People sometimes say that if you have nothing to hide, you have nothing to fear. The legal scholar Daniel Solove has argued that this misunderstands privacy. Privacy is not only about hiding wrongdoing. It protects many interests, and the harm from losing it is often not a single dramatic exposure but the accumulation of many small ones.

• Autonomy and freedom: people think, explore and experiment more freely when they are not observed; constant observation tends to produce conformity.
• Dignity: some aspects of life, such as illness, sexuality and grief, are ours to share or not.
• Relationships: we maintain different relationships by sharing different things with different people; privacy makes intimacy and trust possible.
• Protection from discrimination and manipulation: information about health, finances, beliefs or vulnerabilities can be used to exclude, exploit or persuade people.
• Security: exposed data can be stolen and used for fraud, stalking or harassment.
• Democracy: political participation, journalism and dissent depend on being able to communicate and associate without fear.

Solove also cataloged the different activities that create privacy problems: information collection, such as surveillance and interrogation; information processing, such as aggregation, identification and secondary use; information dissemination, such as disclosure and exposure; and invasion, such as intrusion into a person's life. Data-driven systems raise problems in every category, but aggregation and secondary use are especially important. Aggregation is the combination of pieces of information that are harmless separately into a profile that reveals much more. Secondary use is the use of information for a purpose other than the one for which it was collected, which is exactly what Brightline is asking for.

Inference: privacy beyond what you share

Machine learning adds a further problem: inference. Models can predict sensitive facts that people never disclosed, from data that seems harmless. Researchers have shown that patterns of social media likes can predict personality traits, political views and sexual orientation with accuracy well above chance. Purchase histories can suggest pregnancy or illness. Location data can reveal religion from visits to places of worship, health from visits to clinics and relationships from overnight stays. The person never told anyone; the system inferred it.

Inference challenges the control view of privacy, because you cannot control information you did not know could be derived. It also challenges data protection laws that focus on the data collected rather than on what is learned from it. Kwame admits that his insurer's models use hundreds of variables, and that no one has checked what sensitive traits they might be proxies for. Sabine asks whether a person has a privacy interest in a prediction about them that might be wrong. Varga says the law is unsettled, but that the ethical answer seems to be yes: being treated on the basis of an inferred trait can wrong you whether or not the inference is accurate.

Why removing names is not enough

Brightline promises de-identification: removing direct identifiers such as names and health card numbers. But research has repeatedly shown that de-identified data can often be re-identified by linking it with other information. In the 1990s, the computer scientist Latanya Sweeney estimated that a large majority of Americans could be uniquely identified by just three facts: postal code, date of birth and sex. She demonstrated the point by finding the governor of Massachusetts's medical records in a supposedly anonymous state health data set. In 2008, Arvind Narayanan and Vitaly Shmatikov showed that users in a data set of movie ratings released by Netflix for a research competition could be re-identified by comparing their ratings with public reviews.

Several techniques reduce the risk. Generalization replaces precise values with ranges, such as an age band instead of a birth date or the first three characters of a postal code instead of the full code. Suppression removes rare records or values. A data set is k-anonymous if every combination of identifying attributes is shared by at least k records, so that no individual can be singled out from fewer than k people. Differential privacy, a more recent approach, adds carefully calibrated random noise to the results of queries or to a trained model, so that the output reveals almost nothing about whether any particular individual was in the data. Each technique trades some accuracy for some protection.

# How identifiable is a "de-identified" record? (illustrative records)
from collections import Counter
records = [
    ("R2K", "1978", "F"), ("R2K", "1978", "F"), ("R2K", "1981", "M"),
    ("R2K", "1990", "F"), ("R3B", "1978", "F"), ("R3B", "1978", "F"),
    ("R3B", "1978", "F"), ("R3B", "2001", "M"), ("R2K", "1990", "F"),
    ("R3B", "2001", "M"),
]
groups = Counter(records)
unique = [r for r, n in groups.items() if n == 1]
print("distinct combinations:", len(groups))
print("records that are unique:", len(unique), unique)
print("k for this data set:", min(groups.values()))

distinct combinations: 5
records that are unique: 1 [('R2K', '1981', 'M')]
k for this data set: 1

In the example, the data set is only 1-anonymous: one record is the only person with that combination of postal area, birth year and sex, so anyone who knows those three facts about a person can find their record. Generalizing birth years into decades or suppressing the rare record would raise k. Real data sets have many more attributes, and the free text in records such as ambulance notes is much harder to de-identify than structured fields, which is why Raúl was skeptical.

Consent and its limits

Many privacy laws and practices rely on consent: organizations tell people what they will do with data, and people agree. Consent respects autonomy, and in some situations it is the right basis for using data. But as a general solution it has well-known problems. Privacy policies are long and vague; one widely cited study estimated that reading every policy an average American encounters would take weeks each year. People often have no real choice, because refusing means giving up a service they need. Consent is given once, but data is used for years in ways no one could foresee. And consent by one person can expose others: a person who shares their genetic data with an ancestry company reveals information about their relatives, who never agreed.

These problems have led many scholars to argue that consent should be supplemented by substantive limits: rules about what organizations may do with data regardless of consent, such as prohibitions on some uses, limits on collection to what is necessary for a stated purpose, and duties to protect data and delete it when it is no longer needed. Several of these ideas, known as fair information practices, date back to the 1970s and appear in most modern privacy laws.

Privacy law in the United States

In the United States, the Fourth Amendment protects against unreasonable searches by the government. Since Katz v. United States (1967), courts have asked whether a person has a reasonable expectation of privacy. Under the third-party doctrine, developed in cases such as United States v. Miller (1976) and Smith v. Maryland (1979), information voluntarily shared with a third party, such as a bank or a phone company, was traditionally not protected. In Carpenter v. United States (2018), the Supreme Court limited that doctrine, holding that police generally need a warrant to obtain extended historical cell-site location records, because they reveal an intimate picture of a person's movements.

For the private sector, the United States has no comprehensive federal privacy law. Instead, it has sector-specific laws, such as HIPAA for health information held by health care providers and insurers, the Fair Credit Reporting Act for credit reports, and COPPA for children's data online, together with the Federal Trade Commission's authority over unfair and deceptive practices. Since California's Consumer Privacy Act took effect in 2020, a growing number of states have passed comprehensive privacy laws giving residents rights to access, delete and opt out of the sale of their data. Illinois's Biometric Information Privacy Act, discussed in the next lesson, is one of the strongest state laws. Riverton General's records would be covered by HIPAA, which allows some uses of de-identified data and sets standards for de-identification.

Back to the request

Varga asks the class to apply contextual integrity to Brightline's request. The information is clinical records. It was shared by patients seeking care, with clinicians bound by confidentiality. The proposed flow sends it to a commercial company to improve a product the hospital may buy. Raúl says that flow is not obviously inappropriate: improving emergency care is close to the purpose for which the data was collected, and patients might well accept it. Sabine says that depends on the terms: whether the company can use the data for other clients, whether it can sell it, and whether patients are told. Kwame suggests technical safeguards: let the company train the model inside the hospital's systems without taking the data away, use only the fields needed, and apply differential privacy to the trained model.