cisr gw-tri center for intelligent systems research gw transportation research institute the george...

41
CISR CISR GW-TRI GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, Intelligent Vehicle Systems Symposium Driving Simulator Experiment: Detecting Driver Fatigue by Monitoring Eye and Steering Activity Dr. Azim Eskandarian, Riaz Sayed (GWU)

Upload: emmeline-shaw

Post on 24-Dec-2015

216 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Center for Intelligent Systems Research

GW Transportation Research InstituteThe George Washington University,

Virginia Campus, 20101 Academic Way, Ashburn, VA 20147

NDIA 3rd Annual Intelligent Vehicle Systems Symposium

Driving Simulator Experiment:Detecting Driver Fatigue by Monitoring Eye

and Steering Activity

Dr. Azim Eskandarian, Riaz Sayed (GWU)

Page 2: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Research Objective

Conduct Simulator Experiment and Analyze the Data, to search for a system for automatic detection of drowsiness based on driver’s performance

Page 3: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Significance of the Problem• Drowsiness/Fatigue Related Accident Data:

• NHTSA Estimates 100,000 drowsiness/fatigue related Crashes Annually

• FARS indicates an annual average of 1,544 fatalities

• Fatigue has been estimated to be involved in 10-40% of crashes on highways (rural Interstate)

• 15% of single vehicle fatal truck crashes

• Fatigue is the most frequent contributor to crashes in which a truck driver was fatally injured

Page 4: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

• A drowsy/sleepy driver is unable to determine when he/she will have an uncontrolled sleep onset

• Fall asleep crashes are very serious in terms of injury severity

• An accident involving driver drowsiness has a high fatality rate because the perception, recognition, and vehicle control abilities reduces sharply while falling asleep

• Driver drowsiness detection technologies can reduce the risk of a catastrophic accident by warning the driver of his/her drowsiness

Significance of the Problem

Page 5: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Driver Drowsiness Detection Techniques 1. Sensing of driver physical and physiological phenomenon

– Analyzing changes in brain wave or EEG

– Analyzing changes in eye activity and Facial expressions

• Good detection accuracy is achieved by these techniques

• Disadvantages:

– Electrodes have to be attached to the body of the driver for sensing the signals

– Non-contact type sensing is also highly dependant on environmental conditions

Page 6: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

2. Analyzing changes in performance output of the vehicle hardware

– Steering, speed, acceleration, lateral position, and braking etc.

• Advantages:

– No wires, cameras, monitors or other devices are to be attached or aimed at the driver

– Due to the non-obtrusive nature of these methods they are more practically applicable

Driver Drowsiness Detection Techniques

Page 7: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Approach for Drowsiness Detection and Driver Warning

Page 8: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Experiment• Conducted in the Vehicle Simulator Lab of the CISR.

GWU VA Campus, Ashburn VA.

• Twelve subjects between the ages of 23 and 43

• Test Scenario consisted of a continuous rural Interstate highway, with traffic in both directions Speed limit of 55 mph.

• Morning session 8 – 10 am

• Night session 1 – 3 am

Page 9: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

CISR Driving Simulator

Page 10: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Eye Tracking Equipment

Page 11: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Sample Data From Simulator

RUN# ZONETIME SPEEDLIM CRASHB CRASHV LANEX BRAKEFOR BRAKETAP

1 0 35 0 0 0 0 0

1 2.1 35 0 0 0 0 0

1 4.2 35 0 0 0 0 0

1 6.2 35 0 0 0 0 0

1 8.3 35 0 0 0 0 0

STEERPOS STEERVAR LATPLACE LATPLVAR SPEED SPEEDVAR SPEEDDEV

-0.1 0 -0.09 0 53.71 0 -4.65

0.2 0 -0.22 0 53.71 0 -4.65

0.4 0 -0.31 0 53.71 0 -4.65

0 0 -0.35 0 53.71 0 -4.65

Page 12: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Lateral Position of VehicleD a y 1

- 3

- 2

- 1

01

2

3

T i m e

La

tera

l P

os

itio

n

D a y 4

- 3

- 2

- 10

12

3

T i m e

La

tera

l P

os

itio

n

Page 13: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Power Spectrum Density for Vehicle Lateral Position

0

0.1

0.2

0.3

0.4

0.5

0.6

0 500 100 0 1500 2000

TIME

PS

D

DAY-1

DAY-2

DAY-3

DAY-4

ak

2

k1

n

T

Page 14: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Steering Anglefilter correction for curves

- 10

- 5

0

5

10

15

20

Time

Ste

eri

ng A

ngle

Curvatu reNo Curva ture

Page 15: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Hypothesis• The hypothesized relationship between

driver state of alertness and steering wheel

position is that under an alert state, drivers

make small amplitude movements of the

steering wheel, corresponding to small

adjustments in vehicle trajectory, but under

a drowsy state, these movements become

less precise and larger in amplitude

resulting in sharp changes in trajectory

(Planque et al. 1991).

Page 16: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

A Hybrid Artificial Neural Network Architecture

Wj1

Unsupervised Layer : Clustering Competitive Algorithm

Supervised Layer: ClassificationFeedforward Algorithm

28 X 8

Page 17: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Hybrid Artificial Neural Network Architecture

Unsupervised Supervised

Adaptive Network

W

Input Output

Desired Output Error

Adaptive Network

W

Input Output

Page 18: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

ANN Training for Unsupervised Competitive Layer

1. Initialize the weight vector randomly for each neuron. 2. Present the input vector X(n) .3. Compute the winning neuron using the Euclidean distance

as a metric.

Where Wi = [w1, w2, …. w8]T is the weight vector of

neuron i.

bi is the bias to stop the formation of dead neurons.

Page 19: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

ANN Training Competitive Layer Continued

• N number of time a neuron wins in competitive layer

and are learning constants and o(n) is the outcome of the present competition (=1 if neuron wins & else = 0).

• Ci initially set to small random value

4. Update the weight vector of the winning neuron Wi* only.

5. Continue with step (2) two until change in the weight vectors reaches a minimum value.

Page 20: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

ANN Training Competitive Layer Continued

• The competitive algorithm moves the weight vectors of all the neurons closer to the center of the clusters.

• Each neuron (or set of neurons) of the competitive layer represents a cluster.

• The Output of the neuron is 1 if it wins the competition and 0 if it losses.

• The Output of the Competitive layer is an

n-dimensional binary vector T(n) = [t1, t2, …….., tn]T .

Page 21: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

ANN Training for supervised feed forward layer

• Step 1: Initialize the synaptic weights and the thresholds to small random numbers.

• Step 2: Present the network with an epoch of training exemplars

• Step 3: Apply Input vector X(n) to the input layer and the desired response d(n) to the output layer of neurons. The output of each neuron is calculated as

Page 22: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

ANN Training Continued

Page 23: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

ANN Training Continued

• N = No. of training sets in one epoch = Learning rate parameter = Momentum constant

• Step 5: Iterate the computation by presenting new epochs of training examples until the mean square error (MSE) computed over entire epoch achieve a minimum value. MSE is given by:

Page 24: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

ANN Training Parameters

• Hybrid architecture using an unsupervised clustering algorithm and a classifier (Back propagation learning algorithm in batch mode)

• Tanhyperbolic activation function, with output range from –1 to 1

• Variable learning rate and momentum were used

• Cross validation during training

Page 25: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Input Discretization of Steering Angle

Steering. Angle(deg)

r1 r2 r3 r4 r5 r6 r7 r8

1.1 0 0 0 0 0 1 0 0 -3.1 1 0 0 0 0 0 0 0 -2.2 0 1 0 0 0 0 0 0 0.8 0 0 0 0 1 0 0 0 -1.7 0 0 1 0 0 0 0 0 3 0 0 0 0 0 0 1 0

Algorithm to select r (ranges) for each driver to compensate performance variability between drivers

Discretized steering angle for one driver :

pkk i

4 ri pk

k i -1

4 for i 1 4 (1)

pkk 9 i

4 ri pk

k 8 i

4 for i 5 8 (2)

Page 26: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

• Some drivers are more “sensitive” to vehicle lateral position and make very accurate corrections to the steering for lane keeping while other are less “sensitive” and make less accurate corrections.

• The result is a low amplitude signal (steering angle) for more “sensitive” drivers and relatively high amplitude signal for less “sensitive” drivers.

• Larger values for Pk will make the descritization ranges

wider to accommodate large amplitude while small values will make them shorter for small amplitudes.

• Therefore, same ANN (8-dimensional descritization) can be used

Accounting for Individual Driver Behaviors

Page 27: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Eye closure data is recorded at 60 Hz

Ci = No. of zero’s in 1 second of data

Ci is further discretized according to the following scheme

Input Discretization of Eye closures

Page 28: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Algorithm to select r (ranges) for each driver to compensate eye closure variability between drivers

P values are representative of variability of eye closures (blinking) for each driver

129ifor9

i17k kpir9

i-18k kp

Sample of a few seconds of Discretized Eye closures for one driver :

Input Discretization of Eye closures

E(T) = [e1, e2, e3, e4] Time T sec

Ci

e1 e2 e3 e4

1 4 1 0 0 0 2 7 0 1 0 0 3 0 1 0 0 0 4 18 0 0 1 0 5 0 1 0 0 0 6 1 1 0 0 0 7 6 0 1 0 0 8 1 1 0 0 0

Page 29: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Input Vector

The two vectors are combined to form a 12 dim

vector J(T)

Vector J(T) is summed over 15 sec time interval to

get the input vector X(n)

Page 30: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

X(n) D(n)

x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 SLEEP WAKE

0 0 1 4 8 2 0 0 12 3 0 0 0 1

0 1 0 14 0 0 0 0 9 2 3 1 1 0

2 0 5 4 3 1 0 0 0 1 5 9 1 0

0 0 2 3 9 1 0 0 11 4 0 0 0 1

0 0 0 10 5 0 0 0 11 3 1 0 0 1

0 5 3 6 1 0 0 0 8 3 2 2 1 0

1 4 1 3 4 0 1 1 7 3 2 3 1 0

1 5 2 0 5 1 1 0 10 1 1 3 1 0

Input and Desired Output Vector

Each row represents the sum of discretized input over a selected time interval, e.g., 15 sec.

Page 31: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

ANN Performance During Training

MSE vs Epoch

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 201 401 601 801 1001 1201 1401 1601 1801

Epoch

Ave

rag

e M

SE

Training

Cross Validation

Page 32: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

ANN Test Data• Driving data from 12 subjects available

• 1 subject night session not recorded due to equipment error.

• 1 subject morning data not available, software error.

• Remaining 10 were used for training ANN and testing results,

• NOTE: training data and testing of the ANN were not the same, Testing data selected randomly from the sets not used in the training

Page 33: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Results

Performance SLEEP WakeMSE 0.0550 0.0554NMSE 0.2205 0.2218MAE 0.1259 0.1245Min Abs Error 0.0000 0.0000Max Abs Error 0.9857 0.9806r 0.8851 0.8840Percent Correct 92.3000 93.0000

Actual Totals Network OutputWake Sleep

Wake 193 179 14Sleep 207 16 191Mis-classified

False Alarm

Actual Totals Network OutputWake Sleep

Wake 193 179 14Sleep 207 16 191Mis-classified

False Alarm

Crash Prediction: All crashes that occurred due to driver falling asleep during the experiment were predicted before the crash occurred.

Page 34: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Morning and Night session results

Subject 01 two day driving

0

1

1 11 21 31 41 51 61 71 81 91 101

111

121

131

141

151

161

171

181

191

201

211

221

231

241

251

261

271

281

291

301

311

321

331

341

15 sec time intervals

St + Eye

Drowsy Wake Crash

0

0.2

0.4

0.6

0.8

Eye

Fraction of time eye is closed

0

1

Steering

Morning Night

Page 35: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Morning and Night session results

Subject 02 two day driving

0

1

1 13 25 37 49 61 73 85 97 109

121

133

145

157

169

181

193

205

217

229

241

253

265

277

289

301

313

325

337

349

361

373

385

397

409

15 sec time intervals

St + Eye

Drowsy Wake Crash

0

0.2

0.4

0.6

0.8

Eye

Fraction of time eye is closed

0

1

Steering

Morning Night

Page 36: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Morning and Night session results

Subject 06 two day driving

0

1

1

11 21 31 41 51 61 71 81 91

101

111

121

131

141

151

161

171

181

191

201

211

221

231

241

251

261

271

281

291

301

311

321

331

341

15 sec time intervals

St + Eye

Drowsy Wake Crash

0

0.2

0.4

0.6

0.8

Eye

Fraction of time eye is closed

0

1

Steering

Morning Night

Page 37: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Morning and Night session results

Subject 07 two day driving

0

1

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 103

109

115

121

127

133

139

145

151

157

163

169

175

15 sec time intervals

St + Eye

Drowsy Wake Crash

0

0.2

0.4

0.6

0.8

Eye

Fraction of time eye is closed

0

1

Steering

Morning Night

Page 38: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Morning and Night session results

Subject 08 two day driving

0

1

1 10 19 28 37 46 55 64 73 82 91 100

109

118

127

136

145

154

163

172

181

190

199

208

217

226

235

244

253

262

271

280

289

298

15 sec time intervals

St + Eye

Drowsy Wake Crash

0

0.2

0.4

0.6

0.8

Eye

Fraction of time eye is closed

0

1

Steering

Morning Night

Page 39: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Time Before Crash When the ANN Generated a first Warning

0

0.5

1

1.5

2

2.5

3

3.5

4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Crash No.

Tim

e be

fore

Cra

sh (

min

)

Page 40: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Conclusions• A non-intrusive method of drowsiness detection using

steering data is possible

• A method using ANN is developed and successfully predicts drowsiness (91% Success Rate)

• Method is solely based on driver’s (Vehicle) steering performance

• Same method may be applied to detection of fatigue or other related driver performance

• Further refining and validation of the algorithm is recommended

• Capturing individual driver’s steering while drowsy requires additional research

Page 41: CISR GW-TRI Center for Intelligent Systems Research GW Transportation Research Institute The George Washington University, Virginia Campus, 20101 Academic

CISR CISR GW-TRIGW-TRICISR CISR GW-TRIGW-TRI

Recommended Additional Research• Additional Simulator Experiments

– Validate the Developed Algorithm– Additional Road Conditions– More Diversified Group of Drivers

• Road (Experimental) Tests in an Instrumented Vehicle

• Further Refining the Algorithm Based on the Road Test Data

• Testing of Other Fatigue Related Scenarios• Research on Warning Systems Integrated With This

Detection System