AI and responsibility

Varga begins with a story from the hospital that has not happened yet. Suppose Riverton General adopts Pathfinder. One busy night, a man in his fifties comes in with chest pain and mild symptoms. Pathfinder ranks him as moderate priority, partly because his vital signs are close to normal and partly, it later turns out, because the model learned from historical data in which men of his background were less often admitted to intensive care. The triage nurse is handling three ambulances at once and accepts the ranking. The man waits ninety minutes and has a heart attack in the waiting room. He survives, but with lasting damage. 'Who is responsible?' Varga asks. The seminar spends the whole session on that question.

Questions about responsibility have come up in every lesson so far. This lesson takes them head on. It asks what it means to be responsible, why AI systems make responsibility harder to assign, and what can be done about it. It also asks whether AI systems themselves could ever be responsible agents.

What responsibility means

The word responsibility has several senses, and debates often go wrong by mixing them. Causal responsibility means being a cause of an outcome: the storm was responsible for the flooding. Role responsibility means having duties because of a position: the charge nurse is responsible for the department's patient flow. Moral responsibility, in the backward-looking sense, means being an appropriate target of praise or blame for what happened. Liability is legal responsibility: being required to compensate, being punished or being subject to other legal consequences. Forward-looking responsibility is the duty to bring about or prevent certain outcomes in the future, such as making sure a system is safe.

Since Aristotle, philosophers have identified two main conditions for being morally responsible for an action. The control condition holds that you are responsible only for what you could control; someone forced at gunpoint is not blameworthy in the usual way. The epistemic condition holds that you are responsible only for what you knew or should have known; someone who could not have foreseen a harm is excused, unless their ignorance was itself culpable. AI systems strain both conditions. Developers may not control or foresee how a learning system will behave in new situations. Users may not understand how it works or when it is likely to be wrong.

The problem of many hands

In 1980 the political philosopher Dennis Thompson described the problem of many hands: when many people contribute to an outcome in an organization, it can be difficult or impossible to identify anyone who is responsible for it, because each person's contribution seems too small, too remote or too constrained. In 1996 Helen Nissenbaum, whose work on privacy you met in lesson three, argued that computer systems make the problem worse, and identified four barriers to accountability in computerized society: many hands; the acceptance of bugs as an inevitable part of software; the use of the computer as a scapegoat ('the computer made an error'); and ownership without liability, as when software companies disclaim responsibility for their products.

The Pathfinder case illustrates every barrier. The data scientists at Brightline chose the training data and the target. Their managers set the deadline and the testing budget. The sales team promised the hospital the model was validated. The hospital's administrators bought it and wrote the policy for its use. The clinical informatics team configured it. The triage nurse accepted its ranking. The regulators did or did not review it. Each person can point to someone else. And the model itself, which no one fully understands, is the perfect scapegoat.

The responsibility gap

In 2004 the philosopher Andreas Matthias argued that machine learning creates a responsibility gap. Traditionally, he said, the manufacturer or operator of a machine is responsible for its consequences, because they can predict and control its behavior. But a learning system's behavior is shaped by its training and experience in ways its makers cannot fully predict or control. If no one can predict or control it, then by the usual conditions no one is responsible, and yet harm occurs. Robert Sparrow, discussing autonomous weapons, made a similar argument.

Philosophers have responded in several ways. Some deny that there is a real gap. They argue that people who choose to deploy an unpredictable system in a high-stakes setting are responsible for that choice, just as a person who releases a dangerous animal is responsible for what it does; the uncertainty is a reason for caution, not an excuse. Others accept that there are gaps in blame but argue that we can still assign liability and forward-looking duties, so that someone must compensate victims and fix problems even if no one deserves moral blame. Others argue that the right response is to design systems and institutions so that gaps do not arise, by keeping humans meaningfully in control and making roles and duties clear in advance.

Moral crumple zones

There is an opposite danger to the responsibility gap: blaming the wrong person. In 2019 the anthropologist Madeleine Clare Elish introduced the idea of a moral crumple zone. In a car, the crumple zone is the part designed to absorb the force of a crash and protect the passengers. In complex automated systems, Elish argued, the human operator nearest to the failure often absorbs the moral and legal blame, protecting the integrity of the technology and the organizations behind it, even when the operator had little real control. She pointed to the pilots blamed for accidents involving automated flight systems and to the safety driver in the Tempe crash discussed in the last lesson.

Raúl says the triage nurse in Varga's story is in a moral crumple zone. She was told to use the system, given no time to second-guess it and trained to trust it, but she is the one who will face the inquiry. Sabine replies that the nurse is still a professional with her own duty of care, and that a good nurse should have recognized chest pain in a man of fifty as a warning sign whatever the model said. Varga says both points are right, and that a fair assessment must look at what each person could actually have known and done given the system they worked within.

Cases where responsibility was tested

Several real cases show how responsibility for automated systems has been handled, and mishandled.

• Therac-25: between 1985 and 1987, a radiation therapy machine controlled by software delivered massive overdoses to several patients, some of whom died. Investigations found software errors, the removal of hardware safety interlocks that earlier models had, and a manufacturer that initially insisted the machine could not be at fault. The case became a classic lesson in software safety and in the danger of treating software as infallible.
• The Post Office Horizon scandal: in the United Kingdom, from 1999 onward, hundreds of local postmasters were prosecuted for theft and false accounting on the basis of shortfalls shown by an accounting system called Horizon, which contained errors the Post Office knew about. Many were convicted, bankrupted or jailed. Convictions began to be overturned in 2021, and a public inquiry examined how the organization had treated the system's outputs as reliable and the postmasters as liars.
• The Dutch childcare benefits scandal: in the Netherlands, tax authorities used risk-scoring practices, including an algorithm that treated dual nationality as a risk factor, and wrongly accused thousands of families of fraud, demanding repayment of benefits. Many families suffered severe hardship. The government resigned in January 2021 after a parliamentary report.
• Robodebt: in Australia, an automated program used from 2016 to 2019 calculated welfare overpayments by averaging annual income data, issuing hundreds of thousands of debt notices, many of them wrong. A Royal Commission reported in 2023 that the scheme was unlawful and found serious failures by officials.

Varga points out what these cases have in common. In each, an automated system's outputs were treated as more reliable than the people affected, the burden of proof was shifted onto those people, and organizations resisted evidence of error. None of these systems were advanced AI; several were ordinary software. The ethical failures were human and institutional: treating the computer as a scapegoat and owning a system without accepting liability for it.

A lawyer who trusted a chatbot

In Mata v. Avianca (2023), a lawyer in New York used a generative AI chatbot to research a brief in a personal injury case. The brief cited several court decisions that did not exist; the chatbot had invented them, complete with quotations. When the opposing side and the judge could not find the cases, the lawyer at first stood by them. The federal judge sanctioned the lawyers, emphasizing that using a new tool does not change a lawyer's duty to check the accuracy of what they file. Courts have since dealt with many similar cases, and some require lawyers to certify how they used AI.

Theo says the case shows that responsibility is not as mysterious as the philosophers make it sound: the company or professional who chooses to use a tool answers for it. Kwame agrees for simple cases but says it gets harder when a model is trained by one company, adapted by a second and deployed by a third, each with contracts that push liability onto the others.

Can AI systems be responsible?

Some people have asked whether sufficiently advanced AI systems could themselves be moral agents, responsible for what they do, or moral patients, owed moral consideration. In 2004 Luciano Floridi and J. W. Sanders argued that artificial agents can be moral agents in a limited sense, if they are interactive, autonomous and adaptable at a given level of description, even if they lack mental states. Most philosophers disagree, holding that moral responsibility requires capacities that current systems lack, such as understanding, the ability to respond to reasons in a genuine way, and the capacity to care about what others think of one's conduct. Blaming a system that cannot feel or understand blame seems pointless.

Separately, there have been proposals to give AI systems a legal status, analogous to the legal personhood of corporations, so that they could hold assets and be liable. A 2017 report of the European Parliament floated the idea of 'electronic persons' for sophisticated robots, but it drew strong criticism and was not adopted. Critics argued that it would let manufacturers and owners escape responsibility by shifting it to entities with no assets of their own and no capacity to change their behavior from fear of liability. The question of whether future AI systems could have interests of their own, and so be owed moral consideration, is more open and is taken up briefly in lesson eleven.

Making responsibility work

The class ends by listing practical measures that make responsibility for AI systems clearer before anything goes wrong.

• Name accountable owners: every system in use should have a person or office responsible for its performance, its monitoring and its retirement.
• Document decisions: record why the system was chosen, how it was tested, what its known limits are and who approved its use.
• Keep human judgment real: give operators the time, training, information and authority to override the system, and do not penalize them for doing so in good faith.
• Allocate liability in advance: contracts and laws should make clear who compensates people harmed by a system, so that victims are not left without a remedy.
• Audit and monitor: check outcomes regularly, including for groups, and investigate incidents without looking only for the nearest human to blame.
• Provide remedies: give affected people a way to complain, appeal and be compensated.