Should a self-driving car ever run people over on purpose?
Imagine that you鈥檙e driving through a residential area when your brakes fail. Directly in your path is a group of five jaywalkers. The only place to聽swerve聽is onto the sidewalk, where a pedestrian is waiting for the signal to change.
Who do you run over, the five jaywalkers or the one law-abiding citizen?
Such stark choices are rare, if they occur at all, and, in a world of human drivers they would be made in milliseconds.聽But in a future where cars drive themselves, the choices will be coded in the operating systems of millions of cars,聽highlighting a paradox of a technology that is expected to save countless lives: The cars may also have to be programmed to run people over.
鈥淭here is a common misconception that because it鈥檚 an automatic system it鈥檚 automatically infallible, and will simply聽brake聽in time when a critical situation develops,鈥� says Leon S眉tfeld, the lead author of a聽聽published Wednesday in Frontiers in Behavioral Neuroscience. 鈥淭his unfortunately just isn鈥檛 realistic. A self-driving car is subject to the same laws of physics as a manually driven car.鈥�
Writing code for autonomous vehicles will require聽us to take our moral intuitions 鈥� those nebulous and often contradictory feelings that color our perceptions of human behavior 鈥� and package them into precise instructions for millions of cars we set loose on our roads. That raises what philosophers call Big Questions: Can you quantify morality? Whose set of morals do we use?
Globally there are an estimated聽聽each year, with聽. And in the US, 94 percent of traffic deaths are聽.聽Elimination of human error on our roads would be a boon to public safety.
But before the public is comfortable having software take the wheel, consumers and regulators will need assurances that the cars are programmed with the moral responsibility that comes with a drivers license. This risk-management programming is not just for the one-in-a-million聽聽event where a crash is unavoidable, but for the routine operation of the vehicle.
鈥淚 just don鈥檛 see a lot of these forced-choice scenarios occurring in actual traffic,鈥� says聽Noah Goodall, a researcher at Virginia鈥檚 Department of Transportation who specializes in the ethics of autonomous vehicles. 鈥淭he idea with this kind of work is to figure out how people assign values to different objects.鈥�
In an effort to measure those values, Mr. S眉tfeld, a doctoral candidate at the Institute of Cognitive Science at the University of Osnabr眉ck, Germany, and his colleagues asked 105 participants to don head-mounted聽virtual-reality聽displays that placed them in the driver鈥檚 seat of a virtual car traveling down a two-lane road. A variety of obstacles, including adults, children, dogs, goats, trash cans, and hay bales, were placed in the lanes, and drivers had to pick which obstacle to strike and which one to spare.
The participants were given either one second or four seconds to decide. The one-second trials showed little consistency, suggesting that participants didn鈥檛 have enough time to deliberately choose what to strike. But when the time constraints were eased, a pattern emerged. In the four-second trials, drivers were more likely to spare the lives humans over animals, children over adults, pedestrians over motorists, and dogs over livestock and wild animals.
These consistent聽choices,聽say the researchers, could be used to develop a one-dimensional 鈥渧alue-of-life鈥� scale that could be used to determine whose safety autonomous vehicles should prioritize. Such a scale has an advantage over more sophisticated models, such as those that rely on neural networks, in that it is straightforward and transparent to the public, potentially leading to a quicker acceptance of driverless vehicles.
But a strict hierarchy may not be enough to capture the moral complexity of balancing risks while driving.
鈥淚f human well-being is always a priority, does that mean a self-driving car may not avoid a dog that runs into the聽street,聽if there is an ever so little chance of mild injury to a human in the process?鈥� asks S眉tfeld. 鈥淲e would argue that there needs to be a system聽that is able to make reasonable decisions even in complex situations, and categorical rules often fail this requirement.鈥�
Iyad Rahwan, a professor at the Massachusetts Institute of Technology who researches the ethics of self-driving cars, cautions that no formula will be truly satisfying for everyone.
鈥淭here is too much focus on identifying the correct answer to the rare ethical dilemmas that a car might face,鈥� says聽Professor Rahwan. 鈥淚 think there is no right answer聽in聽an ethical dilemma, almost by definition. Instead, we need to come up with a balance of risks that聽is聽acceptable. We need a social contract that constitutes an acceptable solution to an ethical dilemma that is unsolvable in any objective sense.鈥�
