1

Autonomous Driving Intelligence System

for Safer Automobiles

Hideo InoueKanagawa Institute of Technology (KAIT) , TUAT

Pongsathorn Raksincharoensak Yuichi Saito

Tokyo University of Agriculture and Technology (TUAT)

2018.September 11-12_IPG-Automotive Apply & Innovate 2018

2

1. Motivation and objectives

2. Risk predictive control ; outline of physical model

3. Safety cushion ;

Context-sensitive hazard anticipation based on

driver behavior analysis and cause-and-effect chain study

4. Summary

Table of contents

Academia

Industry

Government

CommunityCitizen

cooperative project Realizing lively and active lifestyle

for elderly people

3

Deterioration in driving ability reduces self-confidence in driving.

However, elderly drivers are highly motivated to improve QOL.

Intelligent driving systems can help to compensate for this deterioration in

physical ability and overcome the fear of driving.

Recognized the warning but did not brake.

Did not recognize the warning and did not brake

The older the driver, the higher the ratio

that could not recognize the warning.

Drivers over age 60 recognized the

warning but did not brake.

0% 20% 40% 60% 100%80%

Ag

e

20 to 39

40 to 59

60 to 69

Recognized the warning and operated brake.

Fig. 1 Reaction of drivers to active safety system

(Experimental study using Toyota Driving Simulator)

Fig. 2 Vehicle necessity

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

1

Can be used anytime, anywhere

Trains and buses cannot be used

Trains and buses limit time

Trains and buses take too much time

Love to drive for fun

Elderly drivers have high motivation to drive.

Motivation and objectives: Overcoming fear of driving

4

Our challenge to enhance safety technologies by foresighted driving

Establish driving intelligence system

to realize risk predictive driving.

Normal

driving

Risk

predictive

driving

Emergency

driving

Accident !!

10s

5s

1s

10ms

Adaptive Cruise Control

Lane Keeping Assist

Autonomous Emergency Brake

Electronic Stability Control

Avoid collision

Put to practical use

Predict and avoid risk

Enhance comfort

Not established

Put to practical use

Research target

5

Autonomous Driving Intelligence System

Motion P

lannin

g

Refe

rence M

aneu

ver

Course

Distance

to obstacle

Blind spotHorizon

camera

Map

LiDAR

radar

IMU

+

Haptic S

ha

red C

ontr

ol

Elderly Drivers

Vehic

le+

Haptic

Controlwith HMI

Intervention

CAN

Emergency avoidance model

Vped V f

D ped

歩行者飛び出し 速度、タイ ミング

走行速度

Potential

Driver

Maneuver

Dynamics

Environment

10 20 30 40 -2 0 20

2000

4000

6000

Y[m]X[m]

Sensor

Fusio

nS

ce

ne

de

tectio

n ・

Ris

k d

efin

itio

n

A:Driving Environment

Recognition System(Lean SLAM)

B:Risk Predictive System(Expert driver model)

C:Shared Control System(Haptic control torque)

Data-driven

AI

6

1. Motivation and objectives

2. Risk predictive control ; outline of physical model

3. Safety cushion ;

Context-sensitive hazard anticipation based on

driver behavior analysis and cause-and-effect chain study

4. Summary

Table of contents

7

Risk prediction model: Defensive driving, object motion prediction, hazard anticipation

Contour of risk potential generated from surroundings and on-road objects

Driving intelligence must predict the possibility of the appearance of pedestrians from

behind an object and determine the optimum safe speed.

Near-miss data High risk

Low risk

Risk prediction model

8

YXUYXUYXU obsroadrisk ,,,

+

YXU road , YXUobs ,

pyJt

p

min*

Risk predictionRepulsive potential form

Motion planningOptimization problem for optimal yaw rate calculation

n

i

ipipipriskpy qYXUJ1

2

,,, )(),(

Optimal yaw rate:

Cost function:

* Risk is determined by environmental states

(road and obstacles)

Risk potential Steering input

Planned vehicle motion

Risks are predicted by environmental states.

Trajectories of expert drivers can be simulated.

Movie④

Movie③

Expert Drivers Standard Deviation Simulation

Risk predicted motion planning

9

Select the deceleration which results in minimum risk path.

Candidate

Candidate

Risk Potential of Collision with pedestrian

Cost Function Balancing Collision Risk and Vehicle Deceleration (Speed-down)

Longitudinal Control : avoid crash with (virtual) pedestrian

10

Risk Potential Based Motion Planning Method

2

2

2

2 )()(exp,

oY

o

oX

oobs

YYXXYXU

X

Y

+ (Xo, Yo)

Ys (way-point)

Xo Yo

2

2

2exp1,

r

sroad

YYYXU

Ys

Y

X X

Y

`Risk Potential of obstacle

Overtaking a parked car scene

(Including a rush-out pedestrian)

Risk Potential of road boundary

11

X

Y

2

2

2

2 )()(exp,

oY

o

oX

oobs

YYXXYXU

X

Y

+ (Xo, Yo)

Ys (way-point)

Xo Yo

2

2

2exp1,

r

sroad

YYYXU

Ys

Y

X

`

X

Y

YXUwYXUwYXU obsoroadrrisk ,,,

Risk Potential of obstacle

Overtaking a parked car scene

(Including a rush-out pedestrian)

Risk Potential of road boundary

Risk Potential Based Motion Planning Method

12

X

Y

Low risk

+ (Xo, Yo)

Ys (way-point)

X

Y

Medium

X

Y

Risk

high

+ (Xo, Yo)

Ys (way-point)

+ (Xo, Yo)

Ys (way-point)

Low Risk Potential High

Near-miss incident DB

YXUwYXUwYXU obsoroadrrisk ,,, YXUwYXUwYXU obsoroadrrisk , ,, YXUwYXUwYXU obsoroadrrisk ,,, wowo

Cyber-Physical System : Data-driven A.I. for intelligent vehicles

13

Hierarchical structure of risk predictive control

Waypoints

Normal

Driving

Course

Intersection

Ref. driving

course

Parked vehicle Pedestrian

Occluded object

Risk Predictive

Driving

Invisible risks potential

Basic automated driving potential

Data-driven AI

Hazard weighting factor

(Pedestrian, bicycle, etc. )

＋

Map

information

Recognition

information

① Basic driving potential to reproduce course collecting of the model driver.

② Obvious risk potential linked with Data-driven AI

to optimal control brock

Virtual objects

Near-miss incident DB

Visible risks potential

14

1. Motivation and objectives

2. Risk predictive control ; outline of physical model

3. Safety cushion ;

Context-sensitive hazard anticipation based on

driver behavior analysis and cause-and-effect chain study

4. Summary

Table of contents

15

Near-miss Incident

16

Traffic Big Data: TUAT Near-miss Incident Database

A

Annotation

Road type (straight/curved)

Intersection type

Traffic participants density

Number of bars

Pedestrian age

….

17

Pedestrian

Conflict point×

Obstacle

Stopping distance

Pedestrian

Conflict point×

Obstacle

Stopping distance Safety cushion

𝑉𝑐𝑎𝑟′𝑉𝑐𝑎𝑟

𝑉𝑐𝑎𝑟

Unsafe driver

Careful driver

𝑉𝑐𝑎𝑟 > 𝑉𝑐𝑎𝑟′

Dcar DpedOccluded area

Occluded area

Adequate adjustments of safety cushion decreases the number of necessary emergency brakes.

Safety Cushion

18

Conflict: too fast driving: high risk

Our motivation is to create adequate safety cushion through machine learning

Techniques by taking advantage of past knowledge/experience of near-miss incident.

Motivation: Adequate Safety Cushion

too slow driving: no driver acceptance

19

Y

X

Pedestrian

carVCollision point

×

Blind area

Parked vehicle

Ygap

Velocity data

න𝑡∗

𝑡2

𝑉𝑑𝑡Dcar (t*) +Dped (t*) =

t* t2

Dcar Dped

Vehicle

Condition for avoiding a crash

Distance to collision point Require distance to stop

τ: System delay

→Time margin indicator to show near-miss incident level

Safety cushion time

Levels of Near-Miss Incident

20

Pedestrian behavior

(Dped, Vped)

Driver behavior (Vcar)

Vehicle dynamics

constraints

t1 t2 t3No road crossing

Caution/ careless driver

Good anticipation/ No-good anticipation driver

Stopping distance [m]

Multiple nonstationary

t3 t2 t1

μ, amax, τ

Vcar, ax

Factors that Reduce Safety Cushion Time: Conflict Pattern

No road crossing Road crossing !!

Safety cushion time represents the result of conflict of these factors.

21

• The Behavior Models were created for each traffic opponent group (vehicle, pedestrian, bicycle).

• The Behavior Models use the quantified context parameter values (fixed, cannot be influenced) and

the quantified driving behavior (can be actively adapted) as input and correlate them with the

incident level (criticality of the incident)

• The Behavior Models describe the relationship between the input parameters and are used to predict

the resulting incident level.

Quantified context parameter

⇒ Risk values

Behavior Model:

Influences of each input parameter and interactions

Prediction of incident level

Quantified driving behavior

⇒ LHP (Long-term Hazard Potential)

Behavior Model Generation

22

Near-miss

incident database

Test scenario

Annotations

Vehicle data

Feature value

Extraction

Preprocessing

LHP

Risk

values

Level identification

Machine learning

Statically processing

SCT

High lev. : 0 < SCT <= 1 sec

Mid. lev. : 1 < SCT <= 2 sec

Low lev. : 2 sec < SCT

Training data

Preprocessing

⇒ Cause and Effect Chain Studies

⇒ Human Err. Analysis

Important Aspects

Context parameter quantification

Driving behavior quantification

23

Proposed index to quantify the driving behavior

Long-term Hazard Potential (LHP)

The index focusing on the speed behaviour from [-10 sec to -4 sec] before the incident.

・ Maximum velocity increases LHP with large portion.

・ High median of the velocity increases LHP

・ Accelerating situation increases LHP.

・ Decelerating situation decreases LHP.

・ Large variance of velocity decreases LHP.[ -10 sec

to -4 sec ]

24

1.0

1.5

2.0

2.5

3.0

0 20 40 60

Velocity at time when the pedestr ian initiated the road crossing [km/h]

LH

P

0

10

20

30

1.0 1.5 2.0 2.5 3.0

LHP

Fre

qu

en

cy

Careful driver Aggressive driver1.85

R =0.81

Speed profile of careful and aggressive drivers is an acceptable trade-off

between desired vehicle velocity and driving safety.

Long-term Hazard Potential

25

Near-miss

incident database

Test scenario

Annotations

Vehicle data

Feature value

Extraction

Preprocessing

LHP

Risk

values

Level identification

Machine learning

Statically processing

SCT

High lev. : 0 < SCT <= 1 sec

Mid. lev. : 1 < SCT <= 2 sec

Low lev. : 2 sec < SCT

Training data

Preprocessing

⇒ Cause and Effect Chain Studies

⇒ Human Err. Analysis

Important Aspects

Context parameter quantification

Driving behavior quantification

26

Context properties type Definition

Static properties Area type Residential area/Urban and business area/Rural area/Other

Road type Other/One way/Both way

Sidewalk type

Intersection type Other/ T and Y types/ 4 type/ More/ Straight

Road width Lanes: other/ 1/ 2/ 3/ 4/ 5 over

Crosswalk Without/ With

Dynamic properties Parked vehicle 0~2/3~5/More

Pedestrian 0~2/3~9/More

Traffic 0~2/3~9/More

Leading vehicle Without/ With

Other properties Time 6:00~10:00/10:00~16:00/16:00~20:00/20:00~6:00

Weather Sunny and cloudy/Rain and Snow

Age of pedestrian Unknown/Elderly/Mature/Young/Child

Qualitative Context Properties

27

sidewalk type

cond.1 cond.2 cond.3 cond.40

20

40

60

Nu

mb

er o

f in

cid

ent

[%]

High

Mid.

Low

sidewalk type

cond.1 cond.2 cond.3 cond.40

50

100

150

Ris

k v

alue

ID: 355291

ID: 389293

ID: 388181

ID: 388072

1 2

3 4

whigh : High level weight

wmid. : Mid level weight

wlow : Low level weight

fi : frequency, Order value∈ [1, 10]

Quantifying the risk level from driving context properties (Annotations)

28

0

1

2

3

4

5

0 1 2 3 4 5

Predicted safety cushion time

Mea

su

red

sa

fety

cush

ion

tim

e

N=109

SCT prediction model by linear regression analysis

29

• Area type: urban area

• Road type: two-way

• Intersection type: T type

• Parked vehicle: high density

• Leading vehicle: without

uncontrollablecontrollable

0.63

sec

4.02

sec

-2.26

sec

-1.16

sec

ID 355961

① Driver is required to decrease the velocity.

② Ref. vel. can be given from the linear regression.

Reference velocity

30

System Framework

TUAT

Near-Miss

Incident

Database

Camera

GPS, map

Motion sensor

Driving

Context

&

Driving

Behavior

Sensing

Driving

Context

Risk Level

&

Driving

Behavior

Evaluation(Speed,

Acceleration,

Yaw rate, etc.)

Desired safe speed

Real Time

Measurement

・・・・

desVSafe Speed Computation

（Worst Case Scenario）Safe Speed

Scene

classification by

traffic opponent

Vehicle

Pedestrian

Bicycle

Context

parameter

quantification

Risk Level value

for each context

parameter value

Driving behavior

quantification

Long-term Hazard

Potential (LHP)

Quantification of

the influence of

context parameters

and driving

behavior on the

incident level

Behavior Models

Motion

planning

(Risk potential)

31

Normal driving Risk predictive drivingEmergency

driving

10ms100ms1s2s5s10s

Car following, Course tracking AEB, PCS

Accident !!

Lane Keeping Control

Cooperative-ACC

Risk Predictive

Driving Intelligence System

Lane Departure Prevention

Autonomous Emergency Braking/

Pre-Crash Safety

Source : Toyota Source : Toyota

LEVEL 1, 2

LEVEL 1

Summary1: Advanced Safety Vehicle with Driving Intelligence

32

FｅｅｄｂａｃｋTravel conditions database Human factor database

Fun to drive Optimum

eco-driving

Accident

avoidance

• Local driving intelligenceADAS with autonomous driving intelligence

• ICT: Info. & communication technology

V2X

Vehicle

• Global driving intelligenceNear-miss data analysis/accident analysis

Enhanced map for dynamic usage

Traffic congestion prediction

etc.

Big data

Summary2: Cyber-physical system for intelligent driving systems

33

Acknowledgement

This study has been conducted as a part of the research project “Autonomous Driving

System to Enhance Safe and Secured Traffic Society for Elderly Drivers” granted by

Japan Science and Technology Agency (JST). Special thanks to the agency providing

the financial support to conduct the research.

In addition, EDI has supported “Context sensitive hazard

anticipation study” through collaboration with TUAT and KAIT.