correlation between eye movements and mouth movements to detect driver cognitive distraction afizan...

CORRELATION BETWEEN EYE MOVEMENTS AND MOUTH MOVEMENTS

TO DETECT DRIVER COGNITIVE DISTRACTION

afizan azman : [email protected]

qinggang meng : [email protected]

eran edirisinghe : [email protected]

BRAIN INSPIRED COGNITIVE SYSTEMS (BICS) 201016TH JULY 2010MADRID, SPAIN

FACTS AND STATISTICS

Department for Transportation (DfT) in UK has reported in 2007, that 92% of passenger travel is by road.

BBC News has been reported that in year of 2008, there were 2,538 people were killed on Britain’s roads

a.azman, q.meng, e.edirisinghe

RESEARCH OVERVIEW

To propose a comprehensive and effective system to detect drivers’ cognitive

distraction in a real time via physiological measurement


RESEARCH OBJECTIVE

To suggest new features for cognitive distraction detection –

lip and eyebrow movements (future work)

To use data analysis approaches/techniques-

Dynamic Bayesian Networks

To use faceAPI toolkit for lip and eyebrow movement detection.


RESEARCH PLAN

collect real-time data on driver visual and cognitive behaviour-modelling process

recognize what the driver is doing (using contextual information such as manoeuvres, actions and states)

predict the alertness level of the driver

design an interface to assist the driver


DRIVING SAFETY ISSUE

Related to distractionDriver state affecting factors

i.Fatigue

ii.Monotony

iii.Drugs

iv.AlcoholDriver trait factors

i.Experience

ii.AgeEnvironmental factors

i.Road environment demands

ii.Traffic demands Vehicle ergonomics


DISTRACTION

VISUAL MANUAL

COGNITIVE


COGNITIVE DISTRACTIONCognitive

produces(output) distraction or causes(input) distraction

situation that might lead or shift a person from putting his attention doing somethingHarder to learn and measure – internal distractionMind off the roadClosely related to visual distractionDelay respond, slow brake, missed traffic light/signboard, unable to stay in a safe distance

COGNITIVE COGNITIVE

DISTRACTION


COGNITIVE DISTRACTION EFFECT

Any types of distraction can undermine-

(a) vehicle control (b) event detection

Fixation concentration= narrowing of the visual field scanned by observer.

Cognitive load on driver affects-

driver eye’s movement and driver’s event detection


DRIVER COGNITIVE MEASUREMENT

available measurements which can be used to measure cognitive workload for drivers:

Performance measure (primary tasks and secondary tasks)-primary is continuous(lane keeping), secondary is non-continuous(looking rare mirror)

Physiological measurement- major organ, available for real time

Rating scales- subjective measurers after activity is completed


NORMAL FACE


THINKING FACE

Images from Google Image


USED FEATURES

Eye movement-blinking, gaze direction, PERCLOS, saccade

Head pose

Pupil diameter

Heart rate


PROPOSED FEATURESFEATURE SUB PARAM PARAM

EYEBROWSRise duration (in ms)

Movement magnitude (in mm)

LIPS

Bright specularityPink lip

Dark apertureColor

MotionPoint 1 (lip corner position)

Point 2 (lip height)Point 3 (lip corner position)

Point 4 (lip height)

Shape/template

EYES

Fixation and Blinking

Movement HeightWidth

Pupil Diameter

Gaze Rotation

MOUTH Movement HeightWidth


AUTOMATED PROCESSData fusion and data mining, both is complementary processes that contribute

automated process. The automated processes are involved with abductive-inductive (learning and discovery) and deductive (detection) process

General properties Implementation

Abduction Create model hypothesis for specific sets of data to explain

that specific set.Mining (discovery of

models)

Induction Extend model hypotheses for representative sets of data to

make a general assertion or explanation

Deduction Apply models to create hypotheses to detect and classify

(explain) the existence of targetFusing (detection)


ALGORITHMS

A few approaches have been used by recent researchers:

Regression- statistical modelling

AdaBoost- for feature selection

SVM- popular technique

BN- popular technique; DBN and SBN


DBN SVM SBNACCURACY Most accurate. Almost

similar accuracy rate with SVM-86.4

Accurate. Almost similar accurate as DBN-85.3

Significantly less accurate-80.7

SENSITIVITY Very good. Can capture more differences in

driver distraction and can generate more

sensitive model

Similar as SBN, but better because SVM accuracy is higher

Similar as SVM

DECISION/RESPOND BIAS

Most liberal Similar as SBN. Similar as SVM

HIT RATIO 93.2 91.1 87.0

FALSE ALARM 30.6 28.9 34.6 (worst)

CONSTRUCTIONAL DIFFICULTIES

Very difficult Average Easy


BAYESIAN NETWORKIs an attractive modelling tool for human sensing. It combines an intuitive

graphical representation with efficient algorithms for inference and learning.

BNs is a reasoning approach which provides a probabilistic approach to inference.

A set of random variables make up the nodes of the network. Variables may be discrete or continuous.A set of directed links or arrows connect pairs of nodes. If there is an arrow from node X to node Y, X is said to be a parent of Y.Each node Xi has a conditional probability distribution P(Xi|Parents (Xi)) that quantifies the effect of the parents on the node.The graph has no directed cycles (and hence, is a directed, acyclic graph, DAG).


DYNAMIC BAYESIAN NETWORK

Attractive modelling for human sensing tool.Probabilistic graphical modelling to do inference and learning.Encode dependencies among variable in an evolving time.Can fuse variety of information with contexual information and expert knowledge. Examples: Kalmann Filter and HMMs.


DYNAMIC BAYESIAN NETWORK

DBN contains several time slices, where at every time slice, the nodes might give a different action as previous time slice.

BNs for time series has the directed arcs and they should flow forward in time and not backward.

sequence of observation {Y} by assuming that each observation depends on a discrete hidden state

X=hidden state variable

Y=observation variable

)|()......|()()( 1121,......2,1 TTT YYPYYPYPYYYP

)|()|()|()(),(2

1111 t

T

tttttt XYPXXPXYPXPYXP


STATIC BAYESIAN NETWORK


DBN MODEL


INITIAL EXPERIMENT


EXPERIMENTAL SETUP

SETUP1

With Audio Task- Lab Setup

This experiment will be conducted with an audio playing to the subject. Subjects are required to listen to a recorded streaming radio on air. Listen to the song and at the same time watching

the video on the screen. The experimenter will ask questions to the subjects. Questions are based on the recorded audio,

recorded video and trigger questions (questions to cognitively distracting the subject)- auditory-based questions, visual-based

questions, conversation-based questions, arithmetic-based questions.


EXPERIMENTAL SETUP

SETUP2

Real environment- The experiment will take place in a real car on a real road:

Put the facelab cameras on the car’s dashboard (real car)A video of the driver driving the car also will be captured

Questions will be asked to the driver. Use the same questionsAre going to use lane change keeping test (LCT)

This setup needs to consider the contextual information.


PEARSON-R CORRELATIONmagnitude of the r-values showed the strength of the relationship between those two variables

0.0 to 0.3 = negligible correlation0.3 to 0.5 = low correlation0.5 to 0.7 = reasonable correlation0.7 or more = good or strong correlation


INITIAL RESULTS


SCATTER PLOT


THE END

THANK YOU


correlation between eye movements and mouth movements to detect driver cognitive distraction afizan...

Documents

drivers cognitive distraction

cognitive load

cognitive workload

driver visual

lip heightpoint

lip heightshapetemplateeye

lip corner positionpoint

cognitive behaviourmodelling