Autonomous systems: vehicles, robots and weapons

A company has applied to run driverless taxis in Riverton, starting with a few downtown neighbourhoods and expanding if things go well. The vehicles have no one behind the wheel. The company's safety report says its cars have been involved in far fewer injury crashes per kilometre than human drivers in the cities where it already operates. Riverton's transport committee has invited public comment, and Varga asks the seminar to prepare a submission. Theo is enthusiastic: human drivers kill thousands of people every year, mostly through distraction, speed and impairment. Mei is uneasy about something she finds hard to name. 'If one of those cars hits someone,' she says, 'who do you even blame?'

This lesson examines systems that act in the physical world with some independence from human control: automated vehicles, robots that work with people, and weapons that can select and engage targets. These systems raise the questions of earlier lessons, about fairness, transparency and privacy, and add new ones about physical risk, delegation of life-and-death choices and human control.

What autonomy means for machines

In ethics, autonomy usually means self-governance: the capacity of a person to set their own ends and act on reasons they endorse. When engineers call a machine autonomous, they mean something much more limited: the machine can perform a task in some environment without continuous human direction. A robot vacuum is autonomous in this sense, and so is a missile defence system that fires at incoming rockets faster than any human could decide. Machine autonomy is a matter of degree and depends on the task and the setting.

For vehicles, SAE International, an engineering standards body, defines six levels of driving automation, from level 0 to level 5. At levels 0 to 2, the human drives and must constantly supervise, even if features such as adaptive cruise control and lane centring help. At level 3, the system drives in certain conditions, but a human must be ready to take over when asked. At level 4, the system drives itself within a defined operational design domain, such as certain streets in certain weather, and does not need a human to take over. Level 5 would drive anywhere a human could. The taxis proposed for Riverton are level 4 vehicles.

The middle levels raise a special problem. Decades of research on aviation and industrial automation show that people are poor at monitoring systems that work well most of the time: attention drifts, skills fade and, when the system suddenly hands control back, the human may not have enough time or awareness to respond. Systems that require a human to supervise but make supervision boring may be more dangerous than either full human control or full automation. This is sometimes called the handoff problem.

The trolley problem and its critics

Discussions of automated vehicles often begin with the trolley problem, a thought experiment introduced by the philosopher Philippa Foot in 1967 and developed by Judith Jarvis Thomson. A runaway trolley will kill five people unless you pull a lever that diverts it onto a track where it will kill one. Most people say you may pull the lever. But in a variant where you could stop the trolley only by pushing a large man off a bridge into its path, most people say you may not, even though the numbers are the same. The cases are used to explore the difference between killing and letting die, and between harming someone as a means and harming them as a side effect.

Applied to automated vehicles, the trolley problem asks how a car should be programmed to act when a crash is unavoidable: should it swerve to save five pedestrians at the cost of its passenger, or protect the passenger at any cost? In 2018 a team led by Edmond Awad published the results of the Moral Machine experiment, an online survey in which millions of people in more than two hundred countries and territories chose what a car should do in such dilemmas. Respondents generally preferred to spare more lives, to spare humans over animals and to spare the young over the old, but preferences differed across cultures.

Many philosophers and engineers think the trolley framing is misleading. Real crashes rarely present clean choices between known outcomes; they involve uncertainty, split seconds and imperfect perception. The ethically important decisions are made long before any crash: how fast the vehicle drives, how cautious it is near schools and crosswalks, how it handles uncertainty about whether an object is a person, where it is allowed to operate and how its risks are distributed. A car designed to drive slowly and yield generously will face fewer dilemmas. And programming cars to sort potential victims by age or other traits, as some Moral Machine respondents preferred, would raise serious concerns about discrimination. In 2017 a German ethics commission on automated driving concluded that in unavoidable accidents, any distinction based on personal features such as age or sex is strictly prohibited.

Risk, safety and who bears it

The strongest argument for automated vehicles is consequentialist: if they crash much less than human drivers, using them will save many lives. The argument depends on evidence, and the evidence is still developing. Companies publish safety data, and independent researchers and regulators analyze it, but comparisons are difficult. Automated vehicles operate in selected areas and conditions, human crash data are incomplete, and serious crashes are rare enough that it takes many millions of kilometres of driving to measure fatality rates with confidence.

# How much driving is needed to show a lower fatality rate? (rough illustration)
human_rate = 1.3        # deaths per 100 million vehicle-km (illustrative)
target_rate = 0.65      # a fleet that is twice as safe
# With rare events, a common rule of thumb is that about 30 expected events give a usable estimate
events_needed = 30
km_needed = events_needed / target_rate * 100_000_000
print("km needed to observe about 30 deaths at the target rate:", f"{km_needed:,.0f}")
fleet = 1000
km_per_vehicle_per_year = 80000
years = km_needed / (fleet * km_per_vehicle_per_year)
print("years for a 1,000-vehicle fleet:", round(years, 1))

km needed to observe about 30 deaths at the target rate: 4,615,384,615
years for a 1,000-vehicle fleet: 57.7

The calculation shows why safety cannot be proved by fatality statistics alone before deployment: a fleet of a thousand vehicles would need decades of driving to observe enough deaths for a confident comparison. Regulators therefore rely on other evidence, such as injury crashes, near misses, simulation, test-track results and safety cases that argue systematically why the system is safe. Sabine notes that this is the Collingridge dilemma again: the evidence that would settle the question can only be gathered by deploying the technology.

Distribution matters as well as totals. A fleet that is safer on average could still shift risk onto particular groups, such as pedestrians in neighbourhoods where it operates, cyclists whom its sensors detect less reliably, or people with disabilities whose movements it predicts poorly. It could also bring benefits that are unequally shared, such as mobility for people who cannot drive. A just policy asks both whether total risk falls and whether the people who bear new risks have a fair say.

Accountability when things go wrong

On the night of March 18, 2018, in Tempe, Arizona, a test vehicle operated by Uber in self-driving mode struck and killed Elaine Herzberg as she walked a bicycle across a road. The U.S. National Transportation Safety Board found that the system had detected her several seconds before impact but repeatedly misclassified her and did not predict her path; the vehicle's own emergency braking had been disabled during automated operation, on the assumption that the safety driver would intervene; and the safety driver was looking at a phone. The board also faulted Uber's safety culture and the lack of state oversight. Uber was not criminally charged; the safety driver later pleaded guilty to endangerment and received probation.

Mei's question, who to blame, has no simple answer in cases like this. Responsibility was spread across the engineers who designed the perception system, the managers who disabled emergency braking, the company that set the safety policies, the regulators who allowed testing, and the human supervisor who was supposed to watch. This distribution of responsibility is the subject of the next lesson. For automated vehicles, many legal scholars argue that liability should shift toward manufacturers and operators when the system is driving, since they control its design and are best placed to make it safe. Courts have begun to hear such cases: in 2025, a federal jury in Florida found Tesla partly liable for a fatal 2019 crash involving its driver-assistance system, a verdict the company said it would appeal.

In Canada, provinces regulate drivers and vehicle use on their roads, and the federal government sets safety standards for vehicles. Ontario has run a pilot program since 2016 that allows testing of automated vehicles on public roads under conditions, and other provinces have their own rules.

Robots that work with people

Robots are increasingly used in warehouses, hospitals, homes and care facilities. Their ethical questions overlap with those of vehicles, such as physical safety, but also raise questions about relationships and dignity. Social robots designed to interact with people, such as robotic pets used with people living with dementia, can reduce agitation and loneliness, and some studies report benefits. The philosophers Robert and Linda Sparrow argued in 2006 that such robots can involve deception, since they invite people to believe the robot cares for them, and that they may be used to replace human contact rather than supplement it. Defenders reply that comfort matters, that people knowingly enjoy fictions such as stories and stuffed animals, and that the question is how robots are used, not whether.

Care ethics, from the second lesson, gives useful questions here. Does the robot support caregivers in giving care, or replace them? Does it respect the dignity of the person cared for, including their privacy, since many care robots record video and audio? Who decides whether it is used: the person, their family or the institution? Raúl says a lifting robot that saves nurses' backs is a different thing from a robot that is supposed to be someone's friend, and the class agrees that the distinction matters.

Autonomous weapons

The most serious questions about autonomy concern weapons. Lethal autonomous weapons systems are generally described as weapons that, once activated, can select and engage targets without further human intervention. Some defensive systems already work this way within narrow limits, such as systems that automatically fire at incoming missiles. The debate concerns broader systems that might identify and attack people.

Arguments for allowing some autonomy in weapons include that machines do not panic, seek revenge or act from fear, and so might comply with the laws of war more consistently than stressed soldiers; that they can react faster than humans in defensive roles; and that they can reduce risks to a state's own personnel. Arguments against include that machines cannot make the contextual judgments the laws of war require, such as distinguishing a surrendering soldier from a combatant; that there may be no one who can properly be held responsible for unlawful killings, a problem the philosopher Robert Sparrow called the responsibility gap; that delegating decisions to kill to machines violates human dignity; and that cheap, autonomous weapons could lower the threshold for war and spread to groups that ignore the law.

Many states and organizations argue for keeping meaningful human control over the use of force. The International Committee of the Red Cross recommended in 2021 that unpredictable autonomous weapons and those designed to target people be prohibited, and that others be strictly regulated. States have discussed the issue for years under the UN Convention on Certain Conventional Weapons, and in December 2023 the UN General Assembly adopted its first resolution on lethal autonomous weapons, expressing concern and calling for further work. There is no treaty specifically regulating them, and major military powers disagree about whether one is needed.