27 Jul 2017 | 15:00 GMT

The Big Problem With Self-Driving Cars Is People

And we’ll go out of our way to make the problem worse

By Rodney Brooks

(/image/MjkyOTE3Nw.jpeg)Illustration: Bryan Christie Design

A purely interpretive problem that

self-driving cars cannot yet solve is that of making sense of the

way people hold themselves and move. For instance, a self-

driving car could not tell what any human driver could take in at

a glance: The couple conversing animatedly near the curb [left]

are not about to wander into traffic. If, however, one person

turns away from the other and in the direction of the street, it

means she’s about to cross [right].

Reading Body Language:

for the fledgling

in1969 never dreamed that networkingtechnology would upend journalism.Nor did anyone guess that cellularcommunication would make peopleignore one another at the dinner table.Early users of email had no idea ofspam. Henry Ford did not foresee thetraffic jam.

The engineers who built routers

ARPANET(http://ieeexplore.ieee.org/document/5432117/?arnumber=5432117)

Technology has

.Sometimes they are large andtumultuous. It is often well worth thetrouble of trying to figure them outahead of time.

unintendedconsequences(https://www.amazon.com/Why-Things-Bite-Back-Consequences/dp/0679747567)

Right now, the new technology with the biggest buzz is the self-driving car. Are there any likelyunintended consequences of the widespread adoption of self-driving cars? You bet there are! I can think oftwo: Such cars will be pariahs, and their owners will act obnoxiously.

Both difficulties will emerge months or perhaps years after truly self-driving cars have been brought tomarket. Before then, engineers have a great deal of work ahead of them to make the cars safer, morecapable, and more foolproof and to convince regulators to allow them onto the roads. These objectives aregoing to take longer than many proponents of automated driving realize or are prepared to admit.

 

I am confident we will eventually get to fully self-driving cars, but my concern is that during trialdeployments we will run into many unexpected consequences that will delay mass deployment for manyyears. As a robotics researcher and entrepreneur, I have made it my business to imagine and visualize howautomation will work in certain environments and situations. I’ve been doing that lately with autonomouscars. What is my conclusion? To paraphrase Bette Davis in the film : Fasten your seat belts.It’s going to be a bumpy ride.

All About Eve

on a moonless night along a country road and heard a car approaching, I’d get off theroad, climbing into bushes if necessary, until the car had passed. I’d do that because I wouldn’t knowwhether the driver had seen me. In such a setting, we willingly give cars the right-of-way.

If I was walking

5 Levels ofAutonomy

Level 0:Human driver

controls all: steering,

brakes, throttle,

power.

Level 1:

But in the daytime, in an urban area, I might step in front of a car at a stop signwithout a second thought. Alternatively, I might linger on the curb a momentwhile indicating that I am about to step off the curb. Or, if I’m behind the wheel, Imight just blow through the intersection, oblivious to the sign. Two questionsarise: If self-driving cars can’t handle such examples of human caprice, how willpeople feel about sharing space with these new aliens? And how much will theperformance of self-driving cars need to be reduced, or otherwise modified, toenable them to share the roads smoothly with cars that are driven entirely orprimarily by humans?

Consider first a residential neighborhood, such as my own part of Cambridge,Mass., where modest houses share the streets with triple-decker apartmentbuildings. The streets are narrow, in many cases one-way, with very few markedpedestrian crossings. People expect to be able to cross a street at any point, butthey know that there is give-and-take between drivers and pedestrians, oftenmediated by subtle cues of eye contact or body language. Cars and people areviewed as equals, quite unlike the situation you’d find on a narrow country road atnight.

In this neighborhood, cars and people interact in three ways. First, on the longermain roads the cars mostly travel without interruption, but there are stop signsmediating access to these through roads from the smaller streets that cross them.People walking along these main roads assume that they, too, have the right-of-way, expecting that drivers who have stopped on a side street will let them walk infront if they are about to step off the curb. Moreover, these people usually wantthe driver to acknowledge their presence before they step in front of the car.

Most functions are

still controlled by the

driver, but a specific

function (like

steering or

accelerating) can be

done automatically

by the car.

Level 2:At least one driver-

assistance system is

automated. Driver is

disengaged from

physically operating

the vehicle (hands

off the steering

wheel AND foot off

the pedal at the

same time).

Level 3:

Second, when people want to cross a street between intersections or on a mainroad without stop signs, they wait for a gap to show between cars. Only then dothey step out cautiously and confirm that the car is slowing down before theymove into the middle of the road. And third, the sidewalks here are narrow, andwhen snow has made them hard or impossible to traverse, people often choose towalk along the roads instead, trying to provide room for the cars to pass butnevertheless expecting the cars to be respectful of them.

Now consider the very different conditions in

, also in Cambridge.It has shops, an area for bars and restaurants (with the upper floors occupied byMIT spin-off startups). There are marked pedestrian crossings and—usually—people cross at those designated places. They do so because the drivers are a littleless civil here, perhaps because there is a larger proportion of people drivingthrough who are not local residents.

Central Square(https://www.google.com/maps/place/Central+Square,+Cambridge,+MA+02139/@42.3652841,-71.1065549,17z/data=!3m1!4b1!4m5!3m4!1s0x89e3775148e0c0e9:0x5219bcb0bdc1c74b!8m2!3d42.3652841!4d-71.1043662)

People step out tentatively into the marked crosswalks and visually check whetheroncoming drivers are slowing down or indicate in some way that they have seenthe pedestrians. It’s easy to see into the cars and get an idea of what the driver ispaying attention to, even at night, given the ample lighting. Pedestrians anddrivers mostly engage in this kind of brief social interaction, and any lack ofinteraction is usually an indicator to pedestrians that the driver has not seenthem. And, this being the Boston area, when such a driver barrels through thecrossing, the pedestrians get angry and yell at the driver.

In yet more hostile areas, such as parts of New York City, pedestrians and driversoften play even more contentious games, such as purposefully avoiding eyecontact so as to force the other party to yield. The upshot is that an autonomouscar able to drive in one area may be poorly equipped to function in another.

The complexity is not limited to contentious behavior, either. In Central Square,many pedestrians “reward” good behavior by drivers. When the main road is busy,getting on or off it can require a lot of patience on the part of a driver. Pedestrianswho have seen signs of such patience will sometimes voluntarily defer to suchdrivers, waving them through.

These are the sorts of nuances that typically elude artificial intelligence. What ifcars trying for full autonomy can’t handle them? The short answer, of course, isthat they will not be able to accommodate pedestrians as smoothly as humandrivers do.

Driver shifts “safety-

critical functions” to

the vehicle under

certain traffic or

environmental

conditions.

Level 4:Fully autonomous

vehicles perform all

safety-critical driving

functions in certain

areas and under

defined weather

conditions.

Level 5:Fully autonomous

system is equal to

that of a human

driver, in every

driving scenario.

This is not just a matter of social nicety. Consider the challenges posed by a snowyday: Cars will have to be able to perceive people walking along—or in—the street,and then they’ll have to make a decision. Should they pass these people, as mosthuman drivers would, or should they follow slowly, avoiding the risk of passing onthe treacherous roads? The latter tactic would slow traffic for both the occupant ofthe driverless car and for any human drivers behind it. Obviously, some of thosehuman drivers would become annoyed at being stuck behind driverless cars.Driverless cars would then be a nuisance.

Even in good weather, an intersection could vex a robotic car. Let’s say the car isstopped at a stop sign on a side street and identifies two people standing at thecorner. These folks might be about to cross, but then again, they could just bechatting. Or maybe it’s a parent and child waiting for the school bus. A humandriver would assess the situation effortlessly. How long should the driverless carwait? And won’t some

such cars by standing at the side of the road and gesticulating as thoughthey’re about to jump off the curb? People don’t try that with human driversbecause there would be repercussions. Driverless cars, on the other hand,wouldn’t be allowed to try to retaliate.

bored jerks try to spoof (/cars-that-think/transportation/self-driving/overly-polite-self-driving-cars-will-brake-for-jerks)

How will a driverless car let you know it has seen you and is trying to figure outwhether you’re about to cross in front of it? It could just inch forward and thenstop if you made a move toward the road. Otherwise, without social interactions,it would be like the case of the dark country road, in which the driverless car hasto be granted the right-of-way over pedestrians and cars with human drivers. Thatwon’t endear them to people, who are unlikely to welcome the idea of driverlesscars that act as if they own the road. So what’s likely to happen is that driverlesscars will be very wimpy drivers, slowing down—and angering—everybody.

Indeed, from the British Department for

Transport predicts that traffic on highways will slow down somewhat because oftimid autonomous systems until some threshold of autonomous density isreached. But I believe that the dynamics of pedestrian interaction will make theproblem much more serious than that.

a report (http://www.telegraph.co.uk/news/2017/01/06/driverless-cars-will-cause-congestion-britains-roads-worsen/)

Consider that there will, for years, be a range of self-driving cars sharing the roadwith pedestrians and human-driven cars. The self-driving cars will themselvesrange from semiautonomous ones, with level-2 or -3 autonomy, to fullyautonomous ones, at levels 4 and 5 [see sidebar, “5 Levels of Autonomy”]. If asemiautonomous car is not playing by the unwritten rules, bystanders willprobably blame the person using the car. But they won’t have that choice if the caris fully autonomous. So in that case, they will blame the car.

Icons: Anders

Wenngren

Illustration: Bryan Christie Design

When snow pushes pedestrians into

the street, cars will have to decide whether to pass them, as

most human drivers would, or to avoid all risk by following

slowly. The latter tactic would slow traffic for both the occupant

of the driverless car and for any human drivers behind it.

Creeping Like Snails:

It’s not hard to see how this could lead to real contempt for cars with level-4 andlevel-5 autonomy. It will come from pedestrians and human drivers in urbanareas. And people will not be shy about expressing that contempt. In private

conversations with me, at least onemanufacturer is afraid that humandrivers will bully self-driving carsoperating with level-2 autonomy, sothe engineers are taking care thattheir level-3 test cars look the same asconventional models.

ofcourse. The flip side of sociallyclueless autonomous cars is theowners of such cars taking theopportunity to be antisocialthemselves.

Bullying can go both ways,

Up from Central Square towardHarvard Square in Cambridge is astretch of Massachusetts Avenue thatmixes residential and commercialbuildings, with metered parking. Oneday I needed to stop at the UPS storethere to ship a heavy package, and asthere were no free parking spots Ifound myself cruising up and down a100-meter stretch as I waited for a

spot to open up. The thought occurred to me that if I’d had a level-4 or -5 self-driving car I could have leftit to do that circling while I dropped into the store. Such is the root of antisocial behavior: convenience forme versus inconvenience for everyone else.

People will be tempted to take many other little shortcuts with their autonomous cars. I’m sure the ownerswill be more creative than I can be, but here are three additional examples:

People will jump out of their cars at a Starbucks to run in and pick up theirorders, leaving them not in legal parking spots but blocking others,knowing that the cars will take care of getting out of the way if some othercar needs to get by. That may well work, but only by slowing everythingdown for other people. And perhaps the owners will be able to set the

tolerance on how uncomfortable things have to get before the cars move. Ican’t see that ending well.Suppose someone is going to an evening event without much parkingnearby. And suppose autonomous cars are always prowling neighborhoodswaiting for their owners to summon them, so it takes a while for anyparticular car to get through the traffic to the pickup location. Then themembers of a two-car family may send one of their cars earlier in the dayto find the closest parking spot that it can, then rely on their second car todrop them at the event and send it home immediately. When the event isover, their first autonomous car is right there waiting for them. The cost isfoisted off on the commons, in the form of a parking spot occupied all day.(Oh yeah, and by the way, with double the greenhouse gases emitted.)In the various suburban schools that my kids went to there was a pickupritual: Mothers, mostly, would drive up just before dismissal time and lineup in the order of arrival; when school let out, the teachers would bring outthe kids, and the parents and teachers would cooperate to get the kids intotheir car seats. Then off would go the cars, one at a time. When the first fewfamilies have fully driverless cars, one can imagine them sending their carsto wait in line first, so that their kids get picked up first and brought home.There will be a contest to see whose robotic car can get to school first.Teachers, too, will be inconvenienced, but people will still try it.

Illustration: Bryan Christie Design

The owners of a self-driving car might order it

to behave in ways that benefit themselves but harm everyone

else. For instance, a concertgoer might send one of his robotic

cars to park near the event well before it takes place or, even

worse, to circle the area for hours, all in order to be in position

to take him home quickly after the event is over.

Jerk by Proxy:

Early on in the transition to driverlesscars, the rich will have a whole newway to alienate the rest of society. Ifyou’re doubtful, take a drive southfrom San Francisco on U.S. Route 101in the morning and see the Teslasspeeding down the left-hand lane.

why I’mskeptical about autonomous cars: TheUnited States and most othercountries haven’t even managed tofully automate their mass-transitsystems. So how are we supposed toachieve the far more difficult task ofcompletely automating cars?

Here’s another reason

True,

driverless train systems inthe world, but they mostly operate invery restricted environments—in theUnited States most are found inairports and span just a few

kilometers of track, all of it completely segregated from the vehicles and people that are outside thesystem. Such systems closely correspond to level-4 autonomy for cars, but in extremely restrictedgeographical environments. Level-5-autonomy trains would move on tracks with level crossings orfunction as streetcars that share space with automobiles and pedestrians. No one is testing level-5 trainautonomy or even proposing to do it.

there are many(https://en.wikipedia.org/wiki/List_of_automated_urban_metro_subway_systems)

Note how much simpler navigation is for a train than for a car. Trains have rails that physically restrictwhere the trains can go. And note further that all train systems are run by teams of specialists. Individualconsumers do not buy and operate trains, yet that is precisely what we are expecting will happen in thecoming market for self-driving cars.

I believe that self-driving cars will be limited, at first, to applications similar to that of today’s self-drivingairport trains. We’ll see autonomous trucks

behind a single human-piloted truck in designated lanes. But once that convoy is offthe highway, we’ll demand that a human driver be behind the wheel in each truck.

convoying (/transportation/advanced-cars/all-aboard-the-robotic-road-train)

And, just like trains in airports, we’ll see level-4 cars driving themselves in limited, pedestrian-freedomains—in garages, for instance, where drivers can drop off cars and let them

with only inches to spare on each side.park themselves (/cars-

that-think/transportation/self-driving/car-park-thyself)

Somewhat later we might see level-4 autonomy for ride-hailing services in limited areas of major cities,with the ride beginning and ending within a well-defined area where well-observed “walk” signals keeppedestrians and cars apart. Some parts of San Francisco might work for this. Indeed,

with such cars, although each still comes with a human minder behind the wheel, whotakes over in case the software fails.

Uber hasexperimented there (/cars-that-think/transportation/self-driving/uber-crash-leads-company-to-suspend-robotaxi-testing)

We might also see level-4 autonomy on some delivery vehicles in dense urban environments. But they willneed to be ultradeferential to pedestrians, as well as avoiding choke points used by other cars during peakcommuting periods.

eponymous law , “we tend to overestimate the effect of a technology in the short

run and underestimate the effect in the long run.”

As  ’s Roy Amara (https://en.wikipedia.org/wiki/Roy_Amara) famously states(http://isen.com/archives/011126.html)

That is where we are today. People are overestimating how quickly level-5 autonomy will come andoverestimating how widespread level-4 autonomy will become in the near future. They see only thetechnical possibilities, not the resistance that will come when autonomous agents invade human spaces,be they too rude or overly deferential.

Certainly this new way of driving will eventually come. It will creep up on us, finally reducing manualdriving to a recreation confined to specialized entertainment zones. The day of the robocar is inevitable,but that day will not come soon.

And f ?Forget about ’em.

lying cars (/aerospace/aviation/autonomous-air-taxis-will-take-off-in-2017-but-wont-go-far)

This article appears in the August 2017 print issue as “The Self-Driving Car’s People Problem.”

About the Author

is the founder and chairman of and the cofounder of . He wrote “

” for ’sspecial report on the in June 2008.

Rodney Brooks (https://rodneybrooks.com/) Rethink Robotics(http://www.rethinkrobotics.com/) iRobot (http://www.irobot.com/) I,Rodney Brooks, Am a Robot (/computing/hardware/i-rodney-brooks-am-a-robot) IEEE Spectrum

singularity (/static/singularity)