Ethical frameworks for technology

Varga opens the second session with a scenario. Brightline Analytics has tested Pathfinder on two years of Riverton General's records. In the test, using the model would have meant that patients with the most dangerous conditions were seen, on average, twelve minutes sooner. But the model also ranked a small number of patients too low, and in the test data three of them had conditions that later turned serious. The hospital must decide whether to use it. 'Before you vote,' Varga says, 'tell me how you would decide. Not what you would decide. How.'

Theo says you add up the outcomes: twelve minutes sooner for thousands of patients against harm to a few. Sabine says you cannot just add people up; each patient has a right not to be put at risk by a system they never agreed to. Raúl says he would ask what a good nurse would do, and a good nurse would not hand her judgment to a machine but would use any tool that helped her see patients better. Kwame says the real question is what rules everyone affected could accept, including the patients the model gets wrong. Varga tells them they have just named four of the main traditions in moral philosophy, and that this lesson is about how each one applies to technology.

Why frameworks matter

Ethical frameworks are systematic accounts of what makes actions right or wrong, or what makes a life or a character good. They are not formulas that produce answers, and serious philosophers disagree about which, if any, is correct. But they are useful for three reasons. First, they bring out considerations you might otherwise miss: a framework built around rights will make you notice who has not consented, and one built around consequences will make you count people who are not in the room. Second, they help you see why people disagree, which is the first step toward resolving or living with a disagreement. Third, when several frameworks point the same way, you can be more confident in your conclusion.

This lesson looks at five frameworks: consequentialism, deontology, virtue ethics, the ethics of care and contractualism. It then looks at how they are combined in practice, through principles and through design methods such as value sensitive design.

Consequentialism

Consequentialism holds that the rightness of an action depends only on its consequences. The best-known version, utilitarianism, developed by Jeremy Bentham and John Stuart Mill in the eighteenth and nineteenth centuries, holds that the right action is the one that produces the greatest total well-being, counting everyone affected equally. Later versions differ on what counts as well-being (pleasure, preference satisfaction or a list of goods such as health, knowledge and friendship), on whether to judge individual acts or general rules, and on how to handle risk and uncertainty.

Consequentialist thinking is natural in technology. Engineers and companies routinely measure outcomes, and cost-benefit analysis, a tool used by governments to evaluate regulations, is a form of it. For Pathfinder, a consequentialist asks how much good the model does, how much harm, and whether there is an option that does better. That includes options the hospital may not have considered: using the model only as a second opinion, using it only at night when staff are stretched, or improving the model before using it.

Consequentialism has well-known difficulties. It can seem to permit sacrificing a few for the benefit of many, which most people find wrong in some cases. It requires predicting consequences, which is hard for new technologies, as the Collingridge dilemma showed. It can be insensitive to how benefits and harms are distributed: a system that helps the majority while consistently harming a minority might score well on total well-being. Some consequentialists respond by giving extra weight to the worst off, an approach called prioritarianism. Others adopt rules, such as respecting privacy, because rules that are generally followed produce better results than case-by-case calculation.

# Pathfinder test: a simple expected-value comparison (illustrative numbers)
patients = 40000            # emergency patients in the two-year test
urgent = 6000               # patients with the most dangerous conditions
minutes_saved = 12          # average minutes sooner for urgent patients
under_ranked = 3            # urgent patients the model ranked too low and who later worsened

print("urgent patient-minutes saved:", urgent * minutes_saved)
print("urgent patients harmed by under-ranking:", under_ranked)
print("harmed per 1,000 urgent patients:", round(under_ranked / urgent * 1000, 2))
# Under the current process, suppose staff also under-triaged some urgent patients
current_under = 9
print("current under-triage per 1,000 urgent patients:", round(current_under / urgent * 1000, 2))

urgent patient-minutes saved: 72000
urgent patients harmed by under-ranking: 3
harmed per 1,000 urgent patients: 0.5
current under-triage per 1,000 urgent patients: 1.5

Kwame points out what the numbers in the example leave out. The comparison only means something if the current process is measured the same way; if nurses currently under-triage nine urgent patients in the same period, the model is better on that measure too. But numbers do not settle whether the three patients the model harmed were harmed in a different way, by a system they could not question, or whether they shared a characteristic, such as age or a rare condition, that would make the harm unfair.

Deontology

Deontology holds that some actions are right or wrong in themselves, not only because of their results. The most influential deontologist, Immanuel Kant, argued that morality is based on reason and that its central principle, the categorical imperative, can be stated in several ways. One formulation says to act only on principles you could will to be universal laws. Another says to treat humanity, in yourself and others, always as an end and never merely as a means. Modern deontological views often speak of rights, duties and constraints: there are things we may not do to people even to produce better outcomes, such as deceive them, use them without consent or violate their dignity.

Deontology brings out considerations that outcome counting can miss. Consent is central: did patients agree to have their care shaped by a model, and should they have to? Deception matters: if a chatbot is designed to seem human, users are being misled even if they enjoy the conversation. Dignity matters: being sorted by a system that treats you as a set of data points may wrong you even if the result is good. Rights to privacy, to an explanation and to human review, which appear in several laws discussed later in the course, have deontological roots.

Deontology has its own difficulties. Duties can conflict: a duty to protect patients may conflict with a duty to respect their privacy. Strict rules can produce bad results in unusual cases. And it can be hard to say what treating someone merely as a means requires when it comes to large systems that affect millions of people indirectly. Most deontologists accept that consequences matter too; the disagreement is about whether they are all that matters.

Virtue ethics

Virtue ethics, which goes back to Aristotle, focuses on character rather than on rules or outcomes. It asks what a person of good character, someone who is honest, just, courageous, temperate and practically wise, would do, and what kind of person an action makes you. Practical wisdom, which Aristotle called phronesis, is the ability to see what matters in a situation and respond well; it cannot be reduced to rules.

Virtue ethics has been applied to technology in two ways. First, it asks what virtues the people who build and use technology need. The philosopher Shannon Vallor, in her 2016 book Technology and the Virtues, argued that rapid technological change requires 'technomoral virtues' such as honesty, self-control, humility, justice, courage, empathy, care, civility, flexibility, perspective, magnanimity and wisdom. Second, it asks how technologies shape character. A feed designed to maximize engagement may cultivate distraction and outrage; a tool that does students' thinking for them may weaken the skills they came to develop. Raúl's question, what a good nurse would do with Pathfinder, is a virtue question: the answer depends on whether using the tool expresses or erodes the judgment and care that make a good nurse.

Critics say virtue ethics gives too little guidance in hard cases, since people of good character can disagree, and that it says more about individuals than about institutions and systems. Defenders reply that good judgment cannot be replaced by rules, and that institutions can be designed to support or undermine virtue.

The ethics of care

The ethics of care, developed by Carol Gilligan, Nel Noddings, Joan Tronto and others from the 1980s, holds that morality is grounded in relationships and in responding to the needs of particular others, especially those who are dependent or vulnerable. It criticizes traditional theories for focusing on abstract, independent individuals and impartial rules, and for neglecting the work of caring, much of it historically done by women and often unpaid or poorly paid.

Care ethics is especially relevant to technologies used in health care, education, elder care and social services, and to technologies that mediate relationships. It asks whether a system supports or replaces the relationships through which care is given: whether a monitoring device lets a nurse spend more time with patients or turns her into a screen watcher, whether a companion robot for an older person supplements human contact or becomes an excuse to withdraw it. It also asks who does the hidden work of making technologies function, such as content moderators, data labellers and the nurses who correct a model's mistakes.

Contractualism and justice

Contractualism holds that right action is action according to principles that people could agree to. In T. M. Scanlon's influential version, an act is wrong if it would be disallowed by principles that no one could reasonably reject. The test protects individuals: a principle that imposes severe burdens on some people for small benefits to many can be reasonably rejected by those who bear the burdens, even if total well-being would be higher. Kwame's question, what rules the patients the model gets wrong could accept, is a contractualist question.

A related tradition, John Rawls's theory of justice, asks what principles people would choose for their society if they did not know what position they would occupy in it, a device he called the veil of ignorance. Rawls argued that they would choose equal basic liberties for all and would permit inequalities only if they benefited the least advantaged. Applied to technology, the veil of ignorance asks you to design a system as if you did not know whether you would be the patient it ranks well or the one it ranks too low, the applicant it approves or the one it rejects.

Applying several frameworks at once

Varga asks the class to apply all five frameworks to the Pathfinder decision and see where they agree and where they diverge.

The table shows a common pattern. The frameworks reach different conclusions in theory, but in practice they often converge on a conditional answer: use the system if it does real good, with human oversight, honesty about its role, attention to who bears its errors and a way to correct mistakes. Where they diverge, the disagreement usually points to a real tension that deserves attention, such as how much risk it is acceptable to impose on a few patients to benefit many.

Principles and design methods

Because the frameworks are abstract, technology ethics often works through mid-level principles. In bioethics, Tom Beauchamp and James Childress proposed four principles in 1979: respect for autonomy, beneficence (doing good), non-maleficence (avoiding harm) and justice. Many AI ethics documents adapt these principles and add explicability, meaning that systems should be understandable and that someone should be accountable for them. By the early 2020s, researchers had counted well over a hundred sets of AI ethics principles issued by companies, governments and organizations. The principles overlap heavily, with transparency, fairness, non-maleficence, responsibility and privacy among the most common.

Critics point out that principles are easy to agree to and hard to apply, that they can conflict, and that companies can adopt them without changing what they do, a practice sometimes called ethics washing. Principles become meaningful when they are tied to specific practices, measurements, responsibilities and consequences for failure.

Value sensitive design, developed by Batya Friedman and colleagues from the 1990s, tries to build ethics into the design process. It combines three kinds of investigation: conceptual, which identifies stakeholders and the values at stake; empirical, which studies how people actually experience the technology and what they value; and technical, which examines how features of the technology support or hinder those values. Related approaches include privacy by design, developed by Ann Cavoukian when she was Ontario's Information and Privacy Commissioner, participatory design, which involves affected people as co-designers, and ethics reviews built into product development.