autonomous vehicles and validation - t&vs · validation tries to answer the question "are...

© Ricardo plc 2019

Autonomous Vehicles

and Validation

2June 2019© Ricardo plc 2019

Validation tries to answer the question

"Are you building the right thing?”

Barry Boehm, Software Engineering Economics, 1981

But…

for an autonomous vehicle what is the right thing?

In general driving isn’t defined - it’s taught.

The First Problem

3June 2019© Ricardo plc 2019

We don’t know what we are trying to validate.

Therefore…

4June 2019© Ricardo plc 2019

Driving is also taught within a context

The context is

350,000 years of human evolution

16+ years of dealing with the real world

AND

The real world is complex!

The Second Problem

5June 2019© Ricardo plc 2019

Example – what is this?

By Beijing Traffic Management Bureau - PDF document GIF diagrams, Public Domain, https://commons.wikimedia.org/w/index.php?curid=34836136

6June 2019© Ricardo plc 2019

Example – what about this?

By Beijing Traffic Management Bureau - PDF document GIF diagrams, Public Domain, https://commons.wikimedia.org/w/index.php?curid=34836136

7June 2019© Ricardo plc 2019

The ODD is large…

Operational terrain

– Road surface, curvature, banking etc.

Environmental and weather

– Rain, wind, visibility, lighting, glare etc.

Infrastructure

– Signs, traffic lights, road markings, tool booths etc.

Rules of engagement

– Road rules and differences between countries/states/cities

In addition…

People (operator, other drivers, pedestrians)

Animals – kangaroos are a real problem…

Everything else!

Operational Design Domain (ODD)

Adapted from: Koopman & Fratrik, How Many

Operational Design Domains, Objects, and Events?

8June 2019© Ricardo plc 2019

Some of everything else…

By William Warby from London, England - Traffic Light Tree, CC BY 2.0,

https://commons.wikimedia.org/w/index.php?curid=5224129

Smart Australia

https://www.adsoftheworld.com/media/outdoor/specsavers_bus_back_crash

9June 2019© Ricardo plc 2019

Groups yet to (fully) deploy self driving cars…

Waymo (Google)

8 million real world miles The Verge July 2018

8 million miles/day simulation The Atlantic Aug 2017

16 billion miles simulated to date

Tesla

1 Jan 2019 estimated 1 billion miles & Lex Fridman Dec 2018

31 Dec 2019 estimated 2.3 billion miles ibid

GM/Cruise

Honda joined Cruise program at $2.5 billion for a 7% shareholding Reuters Oct 2018

How BIG is the Problem?

10June 2019© Ricardo plc 2019

Going to the moon might be easier...

Image: SpaceX

11June 2019© Ricardo plc 2019

Limiting the ODD

Geo-fencing

Excluding conditions (rain, snow, fog)

Works reasonably well – difficult to extend

Requiring the driver to be ready to take over

Mostly works at L2

Won’t work at L3 – L4: humans are bad at monotonous tasks

External human monitoring

Even more boring than being in the car…

Can only deal with one car at a time

Partial Solutions…

12June 2019© Ricardo plc 2019

Define partial safety goals (specifications)

Stay in lane unless it is unambiguously safe to manoeuvre laterally

RSS - Responsibility-Sensitive Safety

1. Do not hit someone from behind.

2. Do not cut-in recklessly.

3. Right-of-way is given, not taken.

4. Be careful of areas with limited visibility

5. If you can avoid an accident without causing another one, you

must do it.

Top Down Solutions – Rules of Engagement

Adapted from: Nilsson, Safe Self-driving Cars: Challenges and

Some Solutions SSS18

Shalev-Shwartz Shammah, Shashua , On a Formal

Model of Safe and Scalable Self-driving Cars

13June 2019© Ricardo plc 2019

Stay in lane unless it is unambiguously safe to manoeuvre laterally.

What do you mean by lane?

Stay in lane unless…

14June 2019© Ricardo plc 2019

What is a lane?

https://imgur.com/a/hAeQI

https://electrek.co/2019/05/20/tesla-fly-guard-rail-funny-balancing-pack/

To see what the drivers sees there is a video at

https://electrek.co/2018/04/03/tesla-autopilot-crash-barrier-

markings-fatal-model-x-accident/

15June 2019© Ricardo plc 2019

Verification is difficult because we can’t define some basic

concepts.

That’s just the start…

Major Issue

16June 2019© Ricardo plc 2019

Why is this so hard?

Perception Planning Execution Actuation

Planning Checker

Execution Checker

Actuation Checker

ACTUATORSSENSORS

Cross check with

safety envelope

Cross check redundant

information

Cross check Against

Feedback

Heurist planning

Algorithmic Software

Algorithmic Software

How do we know what we perceive is real?

ML Systems

Adapted from https://www.slideshare.net/PhilipKoopman1/edge-cases-and-

autonomous-vehicle-safety-sss-2019

Heuristic planning

17June 2019© Ricardo plc 2019

The vehicle shall always come to a halt at a stop sign.

Consider the partial safety goal..

18June 2019© Ricardo plc 2019

Humans

– When they see something they usually know what it is

• but not always…

– But they don’t always see the something

Autonomous Vehicles

- “see” everything

- have no “understanding”

- they “pattern match”

- no pattern, no match

Perception is a BIG problem

Gu, Doian-Gavit & Garg, BadNets: Identifying Vulnerabilities in

the Machine Learning Model Supply Chain arXiv 2019

19June 2019© Ricardo plc 2019

More might be better…

N-version hardware

Multiple processing units homogeneous/heterogenous

M of N voting

N-version software

Value is debateable…

There is some gain, but not as much as expected – human bias

Ensemble of neural nets

Different models

Different data sets

Not always much better e.g. 94% → 96%

Probably difficult to scale…

Can we engineer a solutions?

20June 2019© Ricardo plc 2019

When humans are uncertain they “often” take more care…

Can we use uncertainty to increase safety?

Can we measure uncertainty?

– Intrinsic uncertainty in a classification

– Extrinsic uncertainty over time (e.g. uncertainty in classification)

Extrinsic example - Uber crash in Phoenix Arizona, object classified as

unknown object,

then as a vehicle,

finally as a bicycle

Uncertainty

PRELIMINARY REPORT

HIGHWAY HWY18MH010

21June 2019© Ricardo plc 2019

Invariants

For example - a person has certain proportions

but not on stilts

Adversarial testing

altering images

changing the focus

changing contrast

adding noise

Repeat until?

The “until” part will always be partly arbitrary.

Other ideas…

22June 2019© Ricardo plc 2019

“perceiving” the stop sign is the problem

stopping at the stop sign isn’t the problem

Posted by u/ken3 on reddit

23June 2019© Ricardo plc 2019

A successful test is a test that fails.

Final though…