afatiguedrivingdetectionalgorithmbasedonfacialmotion...

Research ArticleA Fatigue Driving Detection Algorithm Based on Facial MotionInformation Entropy

Feng You12 Yunbo Gong1 Haiqing Tu1 Jianzhong Liang1 and Haiwei Wang 3

1School of Civil Engineering and Transportation South China University of Technology Guangzhou 510640 China2State Key Lab of Subtropical Building Science South China University of Technology Guangzhou China3School of Transportation and Economic Management Guangdong Communication Polytechnic Guangzhou 510650 China

Correspondence should be addressed to Haiwei Wang whw2046126com

Received 8 April 2020 Revised 14 May 2020 Accepted 25 May 2020 Published 15 June 2020

Academic Editor Dongfang Ma

Copyright copy 2020 Feng You et al is is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Research studies on machine vision-based driver fatigue detection algorithm have improved traffic safety significantly Generallymany algorithms asses the driving state according to limited video frames thus resulting in some inaccuracy We propose a real-time detection algorithm involved in information entropy Particularly this algorithm relies on the analysis of sufficientconsecutive video frames First we introduce an improved YOLOv3-tiny convolutional neural network to capture the facialregions under complex driving conditions eliminating the inaccuracy and affections caused by artificial feature extractionSecond we construct a geometric area called Face Feature Triangle (FFT) based on the application of the Dlib toolkit as well as thelandmarks and the coordinates of the facial regions then we create a Face Feature Vector (FFV) which contains all the in-formation of the area and centroid of each FFT We use FFV as an indicator to determine whether the driver is in fatigue stateFinally we design a sliding window to get the facial information entropy Comparative experiments show that our algorithmperforms better than the current ones on both accuracy and real-time performance In simulated driving applications theproposed algorithm detects the fatigue state at a speed of over 20 fps with an accuracy of 9432

1 Introduction

Every year road traffic accidents cause severe damage tohuman health According to the statistics from the WHOfatigue driving is one of the main reasons behind road trafficaccidents [1] National Sleep Foundation points out thatabout 32 of drivers have at least one fatigue driving ex-perience permonth [2] Fatigue driving is a harmful threat tothe driver and other traffic participants Countries all overthe world have made laws to tackle this problem For ex-ample the Chinese Road Traffic Safety Law stipulates thatldquoDrivers are not allowed to drive continuously for more than4 hours and the rest period between every two long-du-ration driving should be no less than 20 minutesrdquo [3] InEurope the law requires that ldquoDrivers should stop and restfor every 45 hours of continuous driving and the rest periodshould be no less than 20 minutesrdquo [3] In the United Statesthe law provision is that ldquoe cumulative maximum dailydriving time must not exceed 11 hours and the continuous

daily rest time must not be less than 10 hoursrdquo [4] Asmentioned above fatigue driving is solely associated withdriving duration It is subjective to determine whether thedriver is in fatigue state or not without sufficient quantifiedindexes and reliable data analysis

According to relevant data heavy road traffic accidentscaused by fatigue driving account for about 50 of all roadtraffic accidents [5] erefore research on fatigue drivingdetection is inevitable e detection algorithms are of thefollowing types

11 Detection Methods Based on Physiology and BehaviorDetection methods based on physiology and behavior arethose that judge the driverrsquos status by installing an intrusivesensor and collecting data that characterizes the driverrsquos phys-iology psychology and driving operations ese detectionmethods include EEG signal detection [6] ECG signal detection[7] pulse beat detection [8] and EMG signal detection [9]

HindawiJournal of Advanced TransportationVolume 2020 Article ID 8851485 17 pageshttpsdoiorg10115520208851485

12 Detection Methods Based on Machine Vision Withdistinctive characteristics of the vehicle motion and thebehaviors of the driver obtained this method assesses thedriverrsquos fatigue status Machine vision-based detection hasbecome the widely used method in fatigue driving detectiondue to its noninvasion and higher accuracy is methodapplies core technologies including face detection eye po-sitioning and fatigue assessment Yan et al [10] used themask to locate the eye position by obtaining the driverrsquosfacial image and used PERCLOS to evaluate the driverrsquosfatigue state is method has better performance on in-dividuals with conspicuous features but the fabrication ofthe mask has a significant influence on the generalizationperformance of the model Niu and Wang [11] divided theface image in the sequence image into nonoverlappingblocks of the same size en they managed to use Gaborwavelet transform to extract multiscale features In order toselect the most recognizable ones they applied AdaBoostalgorithm is method can effectively recognize differentgenders and postures under various illumination conditionsUsing ldquobright eye effectrdquo Bergasa and Nuevo [12] located eyepositionwith active near-infrared light source equipmenteyused finite-state machine to confirm whether the eye is closedey also applied fuzzy system to evaluate the fatigue stateHowever Bergasarsquos algorithm depends highly on hardwarelevel on the other hand the effectiveness of the ldquobright eyeeffectrdquo strictly relies on surrounding light conditions You et al[13] applied the CAMShift tracking algorithm to make thetargeted areas detectable even they were under occlusionenthe eye feature points were obtained according to the specificproportion relationship of the facial organs Finally they usedPERCLOS to determine driver fatigue state

13 Detection Methods Based on Information Fusion Anyfatigue detection method has its advantages and disadvan-tages So comprehensive monitoring of driver fatigue statusby various methods is promising ldquoAWAKErdquo [14] launchedby the European Union is a driving behavior comprehensivemonitoring system It used many sensors such as images andpressures to synthesize the driverrsquos eye movement the di-rection of eyesight steering wheel grip and other drivingconditions en it made comprehensive detection andevaluation Seeing Machines [15] conducts multifeatureinformation fusion by detecting facial features such asdriverrsquos head posture eyelid movement gaze direction andpupil diameter It completed real-time monitoring of driverfatigue status

Although the technology of fatigue detection has madegreat progress it can be better

(i) Physiology-based driver fatigue detections require avariety of additional monitoring devices or equip-ment It would not only reduce comfort duringdriving but also make the collected data costly andvulnerable which has set back the popularization ofthese methods

(ii) If the light condition changes or the driverrsquos face ispartially occluded for example wearing glasses or

sunglasses AdaBoost fails to accurately locate theface position and give the alarm to the driverpromptly

(iii) At present the commonly used algorithms arebased on PERCLOS which judge fatigue by openingand closing state of the driverrsquos eyes However whenthe driverrsquos eyes are too small the algorithms areeasy to misjudge Moreover other fatigue indicatorsare less commonly used due to lower reliability andless robustness

As above literature studies discussed results of thedriving fatigue detection have defects of high intrusion lowrobustness and low reliability erefore we propose afatigue driving detection algorithm based on facial motioninformation entropy e innovations are as follows

(i) We design a driverrsquos face detection architecturebased on the improved YOLOv3-tiny convolutionalneural network and train the network with theopen-source data setWIDER FACE [16] Comparedwith other deep learning algorithms such asYOLOv3 [17] and MTCNN [18] the algorithmbased on the improved YOLOv3-tiny network ismore accurate and simplified It has fewer calcu-lations and thus is easy to transplant to othermobiles

(ii) We used the Dlib toolkit to extract facial featurepoints recognized by improved YOLOv3-tinyconvolutional neural network en we created theFFTafter analyzing the characteristics of the eye andmouth position Next we constructed FFV whichcontains the overall information of the area andcentroid of each FFT We calculate the FFV of eachframe and write it to the database ereby a stateanalysis data set is established In many researchstudies the basis for assessing the state of the driveris the recognition result of a single frame or a fewframes which reduce the accuracy of fatigue drivingdetection Based on the analysis results of a largenumber of consecutive frames we design slidingwindows of driving fatigue analysis to obtain thestatistical characteristics of the facial motion stateerefore the process of driver fatigue can beobserved

(iii) To get rid of the interference that originated fromthe size differences between every FFT we intro-duce the face projection datum plane and apply theprojection principle to extract the motion featurepoints of the faceen based on the motion featurepoints we propose the facial motion informationentropy which quantitatively characterizes thechaotic degree of the motion feature points of theface Accordingly the driverrsquos fatigue state can bejudged At present the commonly used algorithmsare based on PERCLOS [19] which judge fatigue byopening and closing state of the driverrsquos eyesHowever when the driverrsquos eyes are too small thealgorithms are easy to misjudge erefore we

2 Journal of Advanced Transportation

reveal the difference in the motion characteristicsbetween fatigue driving and nonfatigue driving byproposing facial motion information entropy

is paper is divided into the following seven partse first chapter is the introduction In this part we in-troduce the background and research significance of ourfatigue driving detection system and the research statusfrom home and abroad We propose a fatigue drivingdetection algorithm based on facial motion informationentropy with technical innovations In the second chapterwe explain the algorithm in detail e structure of thisalgorithm is a combination of improved YOLOv3-tinynetwork and Dlib toolkit e former captures ROI whilethe latter obtains facial landmarks and creates a fatiguestate data set We make a description of the definition andcalculation method of facial motion information entropywhich is the main index to represent the fatigue state ethird chapter is the experimental analysis Firstly theexperimental environment and data set are introduceden we use qualitative description and quantitativeevaluation to measure face detection and feature pointlocation Finally we evaluate our fatigue driving detectionalgorithm in two directions accuracy and real time efourth chapter is the conclusion which mainly summa-rizes the main work content of this paper and analyzes theshortcomings of the system and the aspects that need to beimproved en we propose the future optimization di-rection and prospect of the algorithm Other sections areData Availability Conflict of Interests Acknowledg-ments and References

2 Methodology

e overall pipeline of our approach is shown in Figure 1e algorithm consists of the following 4 modules

Face Positioning e original data source is the real-time camera video Based on deep learning theory weapply the improved YOLOv3-tiny network to extractsuspected face regions from complex backgroundsFeature Vector Extraction FFT is a geometry area inevery frame that contains facial features Based on thecoordinates of the suspected face region we obtainfacial landmarks with the application of the Dlib toolkitand construct FFV by calculating the area and centroidof the driverrsquos FFTData Set Building According to the FFV extracted in acertain period the driver state analysis data set isestablished in chronological orderFatigue Judgment We design a sliding window as asampler every time it analyzes several sequential FFVswhich match with the related sequential frames byprojecting the FFV on the facial projection datumAfterwards it loops through all FFVs and outputs afacial motion information entropy corresponding tothe current facial feature point set We then comparethe facial motion information entropy with itsthreshold to evaluate the fatigue state of the driver

21 Face Detection Based on the Improved YOLOv3-TinyNetwork Face detection location is the foundation of driverfatigue detection and the accuracy of the results has a greatimpact on the algorithmrsquos performance So accurate andrapid face detection is the fundamental task of the drivingfatigue detection algorithm In the traditional face detectionalgorithm the face features are mostly based on prespecifiedfeatures such as Haar and HOG [20 21] In terms of Haarfeatures Viola and Jones [22] propose a joint Haar featurefor face detection algorithms However image features maylose because of inappropriate face postures dim lightconditions noise interference or a partially occluded facewhich decreases the robustness and reliability of prespecifiedfeature method Recently deep learning theory provides newways for detection and segmentation [23] It can be dividedinto 2 categories one transfers the target detection model toface detection and segmentation process the other is thecascade methods such as MTCNN [24 25] and CascadeCNN [26] Compared with the traditional methods [27] theface detection based on convolutional neural network ex-tracts features autonomously instead of man-made opera-tion With the support of data sets face detectionperformance has been greatly improved

e YOLO [28] (You Only Look Once) model is a fasttarget detection model based on deep learning [29] It is aseparate end-to-end network that turns target detection intoa regression problem Specifically we can replace the slidingwindow in the traditional target detection to the regressionmethod and convolutional neural network (CNN) [30] ismethod of feature extraction is less affected by the externalenvironment and has the advantage of extracting targetfeatures quickly

Inspired by the idea of YOLO model we transform themultiobjective regression into the single target regressionhence reducing the calculation amount en we improveYOLOv3-tiny network to locate suspected face regions

e YOLOv3-tiny network is a simplified version ofYOLOv3 so it has better real time than YOLOv3 It sim-plifies the YOLOv3 feature detection network darknet-53 to7 conventional convolution layers and 6 Max Pooling layersand 1 Up Sample layer e improved network structure isshown in Figure 2 In the figure ldquoDarknetconv2d BN Leakyrdquo(DBL) is the basic component of the network ldquoConvrdquo is theconvolution layer and ldquoLeaky ReLUrdquo is the activationfunction Batch normalization (Batch Norm) is a regulari-zation method that guarantees the algorithm convergenceand avoids overfitting Concat sandwiches a sample layer inthe middle of two DBL Nonmaximum suppression (NMS)is to eliminate the extra facial box and locate the best driverrsquosface suspected area

We consider that the images used for analysis for fatiguedriving contain only one face If the network shows highaccuracy in multiface detection one face detection will bemore accurate So in the YOLOv3-tiny network trainingphase we use the WIDER FACE (Face Detection Data Setand Benchmark) (httpwider-challengeorg2019html)[16] data set as the driving data e WIDER FACE data setincludes 32203 images and 393703 marked faces which isone of the most common face databases e data set

Journal of Advanced Transportation 3

includes different scales poses occlusions expressionsmakeup and lighting as shown in Figure 3

e WIDER FACE data set has the following features

(i) e data set is divided into three types training settest set and verification set which respectivelyaccount for 40 50 and 10 of the data set

(ii) ere are a large number of faces in each imagewhich contains an average of 122 faces

(iii) e data set pictures are high-resolution colorimages

Firstly based on the YOLOv3-tiny network the pictureof theWIDER FACE data set is adjusted to 10 different sizesand every picture is divided into 13times13 grid cells or 26times 26grid cells en we find the location of the driverrsquos face on

the nonoverlapping grid cell and classify it For each gridcell the network outputs B bounding boxes as well as thecorresponding confidence and the conditional probability ofthe driverrsquos face Finally nonmaximal values are used tosuppress redundant bounding boxes e confidence for-mula is given as

score Pr(Object)lowast IOUtruthpred (1)

where Pr(Object) is the probability of the driverrsquos face If theface is included Pr(Object) 1 otherwise Pr(Object) 0IOUtruth

pred is the intersection over union (IOU) of thebounding box to the real box

ere are four basic elements in the YOLOv3-tinynetwork loss function the central error term of thebounding box the width and high error term of the

Driving video

ImprovedYOLOv3-tiny

network Facepositioning Face image Dlib Feature points

location

Face

pos

ition

ing

State analysis data set State analysis data setSliding

windows

Fatig

ue ju

dgm

ent

FFV

Feat

ure v

ecto

r ext

ract

ion

Dat

a set

bui

ldin

g

Fatiguejudgment

H_FFFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

n n ndash 1 i ndash 1n ndash in ndashi + 1 i 1 0 n n ndash 1 i ndash 1n ndash in ndashi + 1 i 1 0

10

11

Figure 1 Algorithm structure diagram where H_F is facial motion information entropy

Conv 32 times 3 times 3Max Pool 2 times 22=gt104 times 104 times 32


Conv 64 times 3 times 3Max Pool 2 times 22

=gt52 times 52 times 64

Conv 128 times 3 times 3Up Sample times 2


Input 416 times416 times 3Conv 256 times 3 times 3Max Pool 2 times 22=gt13 times 13 times 256

Conv 1024 times 3 times 3=gt13 times 13 times 1024









DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL DBL DBL Conv

DBLDBL Conv NMSConcat

Up

Sam

ple

LeakyReLU

BatchNormConv

DBL (Darknetconv2d BN Leaky)

Figure 2 Improved YOLOv3-tiny network structure diagram


bounding box the error term of the prediction confidenceand the error term of the prediction category We managedto use the offline trained YOLOv3-tiny network to extractthe accurate face region for further research

22 Driverrsquos Facial Motion Feature Extraction

221 Face Feature Location Based on the Dlib ToolkitOn the driverrsquos face area located by the improved YOLOv3-tiny network we used the face key point detection modelbased on the Dlib-ml [31] library to extract the fine-grainedfeatures of a driverrsquos face (as is shown in Figure 4(a)) eDlib library contains 68 face key pointse testing principleis applying cascading shape regression to check all the keypoints of the face component

e face detection process is as follows Firstly thefeature of the input image is extracted including the featuresof the face contour eyebrows eyes nose and mouth con-tours Secondly the extracted features are mapped to the facefeature points through a trained regressor at this point aninitial shape of the key points of the human face componentis generated from its original image irdly gradientboosting [32] is used to iteratively adjust the initial shapeuntil it matches with the real shape then the cascaded re-gressor of each stage is calculated with the least-squaremethod

e face key point detection method of the Dlib library isbased on the ensemble of regression trees (ERT) algorithm[29] It uses the regression tree set to estimate the face featurepoints and the speed of calculation is fast e detection of68 key points in each face takes about 1ms Similar to [33]and [34] this cascade regressor method is available eventhough feature points are partially missing in the trainingsample sete iterative algorithm process uses the followingformula

1113954S(t+1)

1113954S(t)

+ Rt h I 1113954S(t)

1113874 11138751113874 1113875 t 1 T (2)

where T is the number of rounds of the regression and 1113954S(t) is

the current shape estimation each regression Rt( ) predictsan increment based on the input images I and 1113954S

(t) that isRt(h(I 1113954S

(t))) e initial shape used is the average shape of

the training data and the update strategy is the GradientBoosting Decision Tree (GBDT) algorithm [32] Every timefor each separate subregion we train a weak classifier whosepredictive value approximates the true value of that sub-region Ultimately the predicted value of the whole region isthe weighted sum of every predicative value

When the driverrsquos face is detected the feature points ofthe face are obtained in real time by the above algorithm asshown in Figure 4(b)

222 Motion State Parameter Extraction As discussedabove drivers get exhausted naturally during driving due tophysiological and psychological state changes At that timethey are in fatigue state Fatigue driving endangers the driverand other traffic participants as it declines the drivingcognitive and driving skills therefore resulting in misper-ception misjudge and misoperation To ensure drivingsecurity and traffic safety the driver must have a clearunderstanding of the driving condition and surroundingroad environments all the time [35] It requires the driver tocontinually adjust the head orientation and the fixationpoint of the eye Compared to nonfatigue driving thedriverrsquos visual field adjustment behaviors change signifi-cantly whether in the early middle or late stages of fatigue[36] e facial motion state such as movement amplitudeand frequency is abnormal

Hence we propose a Face Feature Triangle to charac-terize the driverrsquos facial motion state Based on face featurelocation we defined a Face Feature Triangle (FFT) Asshown in Figure 5 the midpoint of left eye is A the midpointof right eye is B and the midpoint of mouth is C e threepoints consist of the FFT According to the FFT we definethe Face Feature Vector (FFV) as

(a) (b) (c) (d) (e) (f )

Figure 3 WIDER FACE data set diagram


FFV Fx FyS

radic1113872 1113873 (3)

where (Fx Fy) is the midpoint of the FFT and S is the area ofthe FFT According to the plane trianglersquos center of gravity andarea formula Fx Fy S are as shown in the following equation

Fx Ax + Bx + Cx

3

Fy Ay + By + Cy

3

S AxlowastBy minus BxlowastAy + BxlowastCy minus CxlowastBy + CxlowastAy minus AxlowastCy

11138681113868111386811138681113868

11138681113868111386811138681113868

2

(4)

Among them according to Figure 4(a) Dlib face featurepoint positioning and midpoint two-dimensional coordi-nate formula the coordinates (Ax Ay) (Bx By) and(Cx Cy) are defined as

Ax Ay1113872 1113873 p36x + p39x

2p36y + p39y

21113888 1113889

Bx By1113872 1113873 p42x + p45x

2p42y + p45y

21113888 1113889

Cx Cy1113872 1113873 p60x + p64x

2p60y + p64y

21113888 1113889

(5)

where p36 is the coordinate of point 36 in Figure 4(a)As is shown in Figure 6 FFT varies significantly with the

driverrsquos face position therefore the FFV is suitable forcharacterizing the state of facial motion in the fatigue de-tection algorithm

23Driverrsquos Facial FeaturePointsCollection Generally headposture-based fatigue detection algorithms [37] depend onthe characteristics of instantaneous head motions such asnodding to determine whether the driver is in fatigue state Itis challenging to judge fatigue based on a single frame or asmall number of frames and there may even be misjudg-ment erefore it is necessary to study the statisticalcharacteristics of the driverrsquos facial movement state duringfatigue As described in Section 22 to extract the statisticalcharacteristics of facial motion and find the relationshipbetween statistical characteristics and driving fatigue statewe define FFT Since the area of the FFT varies with thedistance between driverrsquos head and the camera in order toget regularized data we apply a face projection datum planemethod As shown in Figure 7 it projects all FFTs to a preset

0

1

2

3

4

5

6

7 8 9

10

11

12

13

14

15

16

1718 19 20

21 2223 24 25

26

27

28

29

3031 3233 34 35

36 37 38394041 42

43 44454647

48 49 50 51 52 5354

55565758

5960

61 62 63 64656667

(a) (b)

Figure 4 Driverrsquos face feature point acquisition based on Dlib (a) Dlib face feature point positioning (b) Face feature point positioning effect

A B

C

Figure 5 Face Feature Triangle (FFT)


projection datum plane and eliminates the interference thatoriginated from the distance difference e area of theprojection datum plane is S0 and projection formula isshown in the following equation

x Fx minuscol2

1113888 1113889lowast

S

S0

1113971

+col2

y Fy minusrow2

1113874 1113875lowast

S

S0

1113971

+row2

(6)

where ldquorowrdquo and ldquocolrdquo are the numbers of rows and columnsof the input images A point (x y) projected onto the datumprojection plane is defined as a feature point of the driverrsquosfacial motion We establish the feature point set of the driverrsquosfacial motion by counting the feature points in frames andthen construct the statistical model of the driverrsquos facialmotion state e experimental results are shown in Figure 8

24 Driver Fatigue State Assessment Model Based on FacialMotion Information Entropy

241 Facial Motion Information Entropy As mentionedabove in nonfatigue state a driver is active to quickly switch

210020001900180017001600150014001300

270260

250240

230220

210200

50 100 150 200300250

350X

Y

Z

LeftNormalRight

(a) (b) (c)

Figure 6 Different facial movement states and FFV differences whereX isFxY is Fy and Z isS

radic ldquoLeftrdquo stands for the left swing of the face

ldquoNormalrdquo stands for normal face posture and ldquoRightrdquo stands for the right swing of the face

S2

S0

S1

Figure 7 Projection schematic


the fixation point and head orientation whereas in theopposite situation the drivers change their head positionmuch more slowly

To compare the difference between frequency and am-plitude of the gaze point and the head orientation in the twodriving states based on the facial motion feature points wecount the set of facial motion feature points under a largenumber of consecutive frames Figures 9(a) and 9(b) showthe set of facial motion feature points under fatigue andnonfatigue conditions respectively

Accordingly compared with the fatigued driving statethe nonfatigue facial motion feature points are more diver-gent and chaotic ldquoA Mathematical eory of Communica-tionrdquo [38] pointed out that any information is redundant andthe redundancy is related to the probability or uncertainty ofeach symbol (number letter or word) in the message at isinformation entropy a concept from thermodynamics Itrefers to the average amount of information after removingthe redundant parts e following equation shows themathematical expression of information entropy

H(X) minus 1113944xisinχ

p(X) logp(X) (7)

Based on the location of facial feature points in Section221 we extract the FFV and establish the state analysis dataset en the facial motion information entropy is definedaccording to the concept of information entropy us theindicator to assess the degree of chaos of the facial featurepoint set is established e calculation method is as follows

(1) Calculate the center point (Fx Fy) of the facialmotion feature point set and N is the number offeature points as is shown in

Fx ΣFx

N

Fy ΣFy

N

(8)

(2) Calculate the Euclidean distance denoted as li fromeach feature point to the center point wherei 1 2 N as shown in

li

Fx minus Fx( 11138572

+ Fy minus Fy1113872 11138732

1113970

(9)

(3) Calculate the mean value and standard deviation ofdistance as is shown in the following equation

μl 1113936

Ni1 li

N

σl

1113936Ni1 li minus μl( 1113857

2

N

1113971

(10)

(4) e interval Ii is defined as equation (11) wherei 1 2 imax imax is defined as equation (12)

Ii (i minus 1)lowastμl

σl

ilowastμl

σl

1113890 1113891 (11)

imax max l1 l2 lN( 1113857

μlσl

+ 1 (12)

(5) According to the distance from each feature point tothe center point the number of distances falling inthe interval Ii is counted as ni

(6) Calculate facial motion information entropy HF(X)as is shown in

HF(X) minus 1113944

imax

i1p xi( 1113857 logp xi( 1113857 p xi( 1113857

ni

N (13)

242 Design of Driverrsquos Facial Motion Information EntropyClassifier Based on SVM As mentioned above when driversfocus well on driving they usually switch the fixation pointand head orientation in order to get a better view of thedriving environments and the facial motion informationentropy is higher On the contrary information entropy ismuch lower under fatigue driving situations We use thetraining set in the open-source dataset YawDD (httpwwwsiteuottawacasimshervinyawning) [39] It contains fatiguedriving data sets of all ages and people of all races includingdifferent genders and facial features It provides videos thatrecord several common driving conditions such as drivingwith glasses speaking and singing while driving evenpretending to be simulating fatigue

SVM [40] is a machine learning model that adopts thestructural risk minimization criterion under the frameworkof statistical learning theory It is a linear classifier modelwith the largest interval defined in the feature space Given atraining data set S (xi yi) i 1 2 N1113864 1113865 on a featurespace xi isin Rd is the ith input sample and yi isin +1 minus1 is thelabel corresponding to xi When yi +1 xi is called apositive sample and when yi minus1 xi is a negative sample

Generally a linear discriminant function f(x) wTxi +

b in a d-dimensional space can distinguish two types of dataand a classification hyperplane can be described as

wlowastT

middot x + blowast

0 (14)

195 210 220200 205 225215190X

2000

2025

2050

2075

2100

2125

2150

2175

Y

Figure 8 Facial motion feature point set


e normal vector wT and the intercept b determine thesuperclass surface function According to the basic idea ofSVM the constrained optimization problem of linear sep-arable support vector machine can be obtained

minwb

J(w) 12w

22

st yi wT middot xi + b( 1113857ge 1 i 1 2 N

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(15)

In the training phase of the driverrsquos face mark box theimproved YOLOv3-tiny is used as the training network andthe training set is applied to detect the driverrsquos face Asdescribed in Section 241 the driverrsquos facial motion infor-mation entropy is calculated based on the positioning in-formation of the Dlib face feature points Among themwhen yi +1 xi is a positive sample indicating that thedriver is in nonfatigue driving state and when yi minus1 xi is anegative sample indicating that the driver is in fatiguedriving state Combined with the constraints of equation(15) the hyperplane parameters wT and b can be calculatedto obtain the driverrsquos facial motion information entropyclassifier

Experiments show that the projection datum area S0 hasdifferent values which will affect the parameters wT and b ofthe driverrsquos facial motion information entropy classifier Inthe experiment S0 is set to 10000

243 Fatigue Judgment Based on Facial Motion InformationEntropy As mentioned above the original image of thedriver was acquired with an in-vehicle camera and theimproved YOLOv3-tiny network was used to detect thedriverrsquos face e face area will be extracted as an inputsubimage and then the Dlib toolkit is used to obtain thefacial feature points of the subimage if the face is detectedin a frame image If not the system will determine that thedriverrsquos head posture is abnormal If it is determined thatthe driverrsquos head posture is abnormal for more than 10

consecutive frames the system will issue an alarm Basedon the face landmarks the FFV is calculated according tothe coordinates of the eye feature points and the mouthfeature points Within a certain number of frames (thenumber of frames set in this paper is more than 1000frames) we count the FFV per frame Considering thatfatigue often generates during driving if directly calcu-lating the facial motion information entropy of all FFVsthe result may be inaccurate In order to improve accu-racy as is shown in Figure 10 the paper sets a slidingwindow to calculate the facial motion information en-tropy in segments on all FFVs e window size is set to1000 and the sliding step size is set to 100 Each time thesliding window slides the 1000 FFVs in the current slidingwindow are obtained first en we can obtain the set offacial motion feature points in the current window Fi-nally the facial motion information entropy HF(X) in thecurrent window is calculated Set ThHF(X) as the judgmentthreshold by training the SVM classifier on the YawDDtraining set If HF(X)ltThHF(X) the judgment is that thedriver is in fatigue state Otherwise the sliding windowmoves to the next position to continue analyzing

e flow chart of fatigue judgment based on facialmotion information entropy is shown in Figure 11

3 Results and Discussion

In order to verify the validity of the algorithm we evaluatedthe performance of the improved YOLOv3-tiny networkwith the public data setsWIDER FACE and YawDD On thisbasis the design comparison experiment is carried out toverify whether the fatigue driving detection algorithm basedon facial motion information entropy is correct

31 Experimental Environment and Data Set e experi-mental platform is the Intel Core i5-8400 with x86 archi-tecture and the CPU clock speed is 280 GHz Graphicscard is GTX1060 with Pascal architecture (CUDA 92

2000

2025

2050

2075

2100

2125

2150

2175Y

195 215210 220190 200 225205X

(a)

195 215210 220190 200 225205X

2000

2025

2050

2075

2100

2125

2150

2175

Y

(b)

Figure 9 Different drive state facial motion feature point set Facial motion feature point set in (a) fatigue and (b )nonfatigue


CUDNN 72) e RAM is 8G DDR4 and the opencv346image library is used e deep learning computingframework is PaddlePaddle15 e environment of theprogram is python 36 Hardware configuration is shown inTable 1

e data set used in the experiment included the publicdata sets WIDER FACE and YawDD where the public dataset WIDER FACE includes 32203 pictures and 393703marked faces which is used to train Yolov3-tinyrsquos facenetwork However the WIDER FACE data set only containsmarker face images and does not provide any informationabout the driverrsquos fatigue status erefore the WIDERFACE data set cannot be used to analyze driver fatiguestatus YawDD is a data set of fatigue driving detectionincluding male and female volunteers in the naked eyewearing glasses normal state speakingsinging and simu-lated fatigue So we choose YawDD data set as test set offatigue driving detectione detection result of the YawDDdata set is shown in Figure 12

32 Face Detection and Feature Point Location

321 Qualitative Description In order to verify the effec-tiveness of face detection based on the improved YOLOv3-tiny network and the accuracy based on the Dlib facialfeature point location the experiments were performed inthe laboratory and in the vehicles

FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

nn ndash 1i ndash 1 n ndash i n ndashi + 1i10

Figure 10 Sliding windows

Start

Video stream

Improved YOLOv3-tinyface detection

Facedetected

Feature points location

Feature pointslocation

N

Y Y N

N

Y

N Y

Next frame

Calculate FFV

Video end Sliding windows

FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N

Figure 11 Driver fatigue state assessment model flow chart

Table 1 Hardware configuration table

Type Specific parameters

Processor Intel(R)Core(TM)i5-8400 CPU280GHz281GHz

GPU NVIDIA GeForce GTX1060 6GBComputer version Windows 10RAM 800GBPython version 36Opencv version 346Paddle version 15CUDA version 92CUDNN version 72


In the laboratory the light is uniform and does notdrastically change e face recognition algorithm based onimproved YOLOv3-tiny network can accurately detect facesfrom test videos e face area can be correctly marked as isshown in Figures 13(a) and 13(b) (1-1) and (1-2) Besides thealgorithm can detect the driverrsquos face area and mark featurepoints even in the cases of wearing glasses (as shown inFigure 13 (2-1)) head tilting (as shown in Figure 13 (1-3))and expression changing (as shown in Figure 13 (2-2))

In the vehicle experiment the change of illuminationmay cause high interference to the driverrsquos face detectionand feature point location So it is crucial to verify theeffectiveness of the algorithm in the real vehicle scenario Inthe real driving scene the algorithm can complete facedetection and feature point location in case of uneven il-lumination as is shown in Figure 13 (4-1) It can be seen thatthe algorithm has excellent recognition performance androbust performance in both the laboratory and real vehicleand this will provide the basis for the driverrsquos fatigue featureextraction and fatigue state assessment

322 Quantitative Evaluation e improved YOLOv3-tinynetwork provides face landmarks for fatigue driving de-tection Its performance represents the effectiveness of thefatigue driving detection algorithm erefore we quanti-tatively evaluate of the performance of the improvedYOLOv3-tiny network on the WIDER FACE data set

In this paper we adopt the ROC curve [41] theory forevaluation Accuracy is the ratio of the number of correctlypredicted samples to the total number of samples and it isan intuitive evaluation index of model performanceHowever the accuracy rate is difficult to express the prosand cons of the model in case of uneven distribution ofpositive and negative sample data e sensitivity indicatesthe proportion of all positive samples correctly detectedSpecificity indicates the proportion of all negative samplescorrectly detected e ROC curve is a comprehensiveindicator formed by the combination of sensitivity andspecificity and reflects the sensitivity and specificity ofcontinuous variables

(1) Accuracy (ACR) In the task of the driverrsquos face detectionthe ACR is the ratio of the number of correctly detectedimages to the total number of images

ACR Ndetected

Ntotal (16)

where Ndetected is the number of correctly detected imagesand Ntotal is the total number of images

In the process of improving the YOLOv3-tiny networktraining and verification the intersection ratio parameter(IOU) [42] is introduced to measure the similarity be-tween the face detection area and the marked real areaIOU is a standard for measuring the accuracy of a cor-responding object in a specific data set In Figure 14face d is the face area detected by the model face is thereal area marked and the calculation formula is given inthe following equation (17) where Area(face dcapface) isthe area of face dcapface and Area(face dcupface) is the areaof face dcupface

IoU Area(face dcap face)Area(face dcup face)

(17)

e intersection ratio indicates the degree of overlapbetween the model prediction area and the real area As canbe seen from Figure 14 the higher the value is the higherthe detection accuracy is In the case where IOU 1 theprediction box overlaps with the real box Generallyspeaking the object is correctly detected when the IOU ismore than 05 In the face detection process we adopt ahigher threshold In this paper when the IOU is more than075 the face is considered to be correctly detected Fig-ure 15 shows the accuracy curve of the driverrsquos face de-tection during the training of the improved YOLOv3-tinynetwork It can be seen that with the increase of trainingrounds the accuracy of face detection gradually increasese improved YOLOv3-tiny network has an accuracy rateof 985

(2) ROC Curve Sensitivity and specificity are importantevaluation indicators of the pattern recognition model If

Eye open Fps 248

Face yes Mouth close

(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)

Figure 12 e detect result of YawDD data set


you use TP TN FP and FN to indicate the number of true-positive true-negative false-positive and false-negativesamples respectively in a test then the definitions ofsensitivity Sn and specificity Sp are

Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)

Figure 13e results of face detection and feature point location (a) (1-1) (b) (1-2) (c) (1-3) (d) (1-4) (e) (2-1) (f ) (2-2) (g) (2-3) (h) (2-4) (i) (3-1) (j) (3-2) (k) (3-3) (l) (3-4) (m) (4-1) (n) (4-2) (o) (4-3) (p) (4-4)

Face_d cap face

Face_d

Face

Figure 14 Intersection over union


A ROC curve is a graph of the relationship between thetrue-positive rate (sensitivity) and the false-positive rate(1minus specificity) e ROC curve is one of the comprehensiveindicators for characterizing the accuracy of pattern rec-ognition tasks and the closer the ROC curve is to the upperleft corner the better the model performance is

Figure 16 shows the ROC curve of the driverrsquos facedetection model As can be seen from the figure the ROCcurve corresponding to the improved YOLOv3-tiny networkis close to the upper left corner of the graph indicating highaccuracy in face detection

In summary by evaluating the performance of theimproved YOLOv3-tiny network on the WIDER FACE dataset it is shown that the improved YOLOv3-tiny network inthis paper has high accuracy Besides the ROC curve in-dicates that the algorithm can effectively avoid two types oferrors in the driverrsquos face recognition that is to ensure thatthe driverrsquos face can be correctly detected while avoiding themisjudgment on the face

33 Fatigue State Evaluation

331 Accuracy We use the YawDD data set to test theperformance of fatigue detection Face detection and facialfeature point location are the basis of fatigue driving de-tection e FFV of each frame in the on-board video iscalculated and stored based on the facial feature pointsCalculate the FFVs of all video frames in a certain periodand establish a state analysis data set e sliding window(discussed in Section 243) is applied to the state analysisdata set to calculate the facial motion information entropyfor each sliding If the entropy does not exceed the thresholdwe can conclude that the driver is in fatigue state Videos arerandomly selected from the data set for fatigue drivingdetection e process of fatigue driving detection is shownin Figure 11

In this paper we randomly select ten videos from theYawDD test set including nonfatigue driving status andfatigue driving status e facial information entropythreshold for judging fatigue state is 132 and the results areshown in Table 2 It can be seen that the accuracy of thefatigue driving detection in the randomly selected ten videosis 90 and the correct rate of the system in the entire test setof YawDD is 9432

332 Speed Based on hardware configuration as shown inTable 1 a comparison test is performed on the image sourceto verify the real-time performance of the systeme resultsare shown in Table 3

Table 3 illustrates that YawDD Video excels at facedetection time One possible reason is the difference between

0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps

YOLOv3-tiny ACRYOLOv3-tiny final ACR

10

09

08

07

06

05

04

03

02

01

00

ACR

0985

Figure 15 Driver face detection accuracy

ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve


the data reading methods and the YawDD Video methodgets the data from the video stream directly

Our algorithm shows that the system has good accuracyand high-speed performance under various conditions andcan accurately judge the fatigue state of the driver Com-pared with AdaBoost +CNN and CNN+DF_LSTM algo-rithms [43 44] our method improves the accuracy of thefatigue driving detection algorithm It also has better real-time performance which meets the requirements of thefatigue driving detection system e comparative result isshown in Table 4

4 Conclusions and Future Directions

With the rapid increase of global car ownership road trafficaccidents have become one of the leading causes of humandeath in the world Fatigue driving is one of the main causesof road traffic accidents Fatigue driving can seriously affectdriving skills and seriously threaten drivers and other trafficparticipants At present fatigue driving detection and earlywarning have achieved better research results but they stillneed some improvements such as high intrusiveness poordetection performance in complex environments andsimple evaluation indicator erefore we propose a newdetection algorithm for fatigue driving based on facialmotion information entropy e main contributions are asfollows

(i) We design a driverrsquos face detection architecturebased on the improved YOLOv3-tiny convolutionalneural network and train the network with theopen-source data set WIDER FACE Compared

with other deep learning algorithms such asYOLOv3 [17] and MTCNN [18] the algorithmbased on the improved YOLOv3-tiny networkimproves the face recognition accuracy simplifiesthe network structure and reduces the amount ofcalculation en it is more convenient to trans-plant to the mobile e accuracy rate of face rec-ognition based on the improved YOLOv3-tinynetwork is up to 985 and single test just takes3452ms

(ii) e Dlib toolkit is used to extract facial featurepoints on the face area that is located by the im-proved YOLOv3-tiny convolutional neural net-work en the driverrsquos FFT is established byanalyzing the positioning characteristics of the eyeand mouth Finally the driverrsquos FFV is constructedby the area and centroid of FFT We calculate theFFV of each frame and write it to the databaseereby a state analysis data set is established Inmany research studies the basis for assessing thestate of the driver is the recognition result of a singleframe or few frames which reduce the accuracy offatigue driving detection In this paper based on theanalysis results of a large number of consecutiveframes we design sliding windows of driving fatigueanalysis to obtain the statistical characteristics of thefacial motion state erefore the process of driverfatigue can be observed

(iii) To eliminate the interference of change of the FFTrsquosarea to fatigue driving judgment we introduce theface projection datum plane and apply the projec-tion principle to extract the motion feature points ofthe face en based on the motion feature pointswe propose the facial motion information entropywhich quantitatively characterizes the chaotic de-gree of the motion feature points of the face enwe train the SVM classifier using the open-sourcedata set YawDD [37] Experiments show that the

Table 2 Sample fatigue test table

Sample number Facial motion information entropy Actual driving status Predictive driving status1 [123 096 056 120 140 049 065 045 075] Fatigue Fatigue2 [110 142 086 052 097 095 150 088] Fatigue Fatigue3 [250 242 265 193 201 289 332 321] Nonfatigue Nonfatigue4 [057 087 034 067 095 112 121 129 101] Fatigue Fatigue5 [198 187 193 203 323 342 334 272] Nonfatigue Nonfatigue6 [062 057 088 102 142 145 092] Fatigue Fatigue7 [222 152 233 2 78 311 207 298 304] Nonfatigue Nonfatigue8 [135 102 122 078 056 022 024 031 055] Fatigue Fatigue9 [244 257 272 198 142 130 223 289 266] Nonfatigue Fatigue10 [150 089 076 071 065 088 031 042 051] Fatigue Fatigue

Table 3 e time spent in fatigue status judgment

Image source Face detection time (ms) Facial feature point positioning time (ms) Calculate FFV time (ms) Total time (ms)Camera 3452 1391 1 4943YawDD Video 3213 1391 1 4704

Table 4 Comparison of fatigue detection algorithms

Algorithms Accuracy () Speed (msmiddotfminus1)AdaBoost +CNN 9210 5861CNN+DF_LSTM 9148 6564Algorithm in this paper 9432 4943


projection datum area S0 has different values whichwill affect the parameters wT and b of the driverrsquosfacial motion information entropy classifier Wedesign fatigue judgment algorithm based on facialmotion information entropy and the comparisonexperiments show that our algorithm has an ac-curacy rate of 9432 and an algorithm speed of4943msf which further improve the accuracy andspeed of the driverrsquos fatigue detection algorithm

In the future we will focus on the following research

(1) Upload the results of the fatigue detection to thecloud platform and combine the big data analysistechniques to analyze the driverrsquos fatigue period [45]

(2) Integrate the fatigue driving detection algorithm intoADAS (Advanced Driving Assistant System) [46 47]

(3) Expand the applicable environment of the algorithmand explore the driver fatigue detection algorithmbased on facial motion information entropy in nightenvironment [48 49]

Data Availability

e data used to support the findings of this study areavailable from the first author and the corresponding authorupon request

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this article

Acknowledgments

is work was supported in part by the National NaturalScience Foundation of China (Grant no 51808151) Guang-dong Provincial Public Welfare Research and CapacityBuilding Special Project (Grant no 2016A020223002) SouthChina University of Technology Central University FundProject (Grant no 2017ZD034) Guangdong Provincial Scienceand Technology Plan Project (Grant no 2017A040405021) theFundamental Research Funds for Guangdong CommunicationPolytechnic (Grant no 20181014) Guangdong ProvincialNatural Science Foundation (Grant no 2020A151501842)Guangzhou 2020 RampD Plan for Key Areas (Grant no202007050004) and by State Key Lab of Subtropical BuildingScience South China University of Technology (Grant no2020ZB20)

References

[1] A Amodio M Ermidoro D Maggi S Formentin andS M Savaresi ldquoAutomatic detection of driver impairmentbased on pupillary light reflexrdquo IEEE Transactions on Intel-ligent Transportation Systems vol 20 no 8 pp 3038ndash30482019

[2] X Li X Lian and F Liu ldquoRear-end road crash characteristicsanalysis based on Chinese in-depth crash study datardquo inProceedings of the 16th COTA International Conference ofTransportation Professionals Green and Multimodal

Transportation and Logistics pp 1536ndash1545 Shanghai ChinaJuly 2016

[3] F Chen and S Chen ldquoInjury severities of truck drivers insingle- and multi-vehicle accidents on rural highwaysrdquo Acci-dent Analysis amp Prevention vol 43 no 5 pp 1677ndash1688 2011

[4] X Zhu Z Dai F Chen X Pan and M Xu ldquoUsing the visualintervention influence of pavement markings for ruttingmitigation-part I preliminary experiments and field testsrdquoInternational Journal of Pavement Engineering vol 20 no 5pp 734ndash746 2019

[5] R Zhang F You X N Chu L Guo Z-C He andR-B Wang ldquoLane change merging control method for un-manned vehicle under V2V cooperative environmentrdquo ChinaJournal of Highway and Transport vol 31 pp 180ndash191 2018

[6] YWang X Liu Y Zhang Z Zhu D Liu and J Sun ldquoDrivingfatigue detection based on EEG signalrdquo in Proceedings of the5th International Conference on Instrumentation and Mea-surement Computer Communication and Control pp 715ndash718 Qinhuangdao China September 2015

[7] R Bhardwaj P Natrajan and V Balasubramanian ldquoStudy todetermine the effectiveness of deep learning classifiers forECG based driver fatigue classificationrdquo in Proceedings of the13th International Conference on Industrial and InformationSystems pp 98ndash102 Punjab India December 2018

[8] M K Sharma and M M Bundele ldquoDesign amp analysis of k-means algorithm for cognitive fatigue detection in vehiculardriver using oximetry pulse signalrdquo in Proceedings of the IEEEInternational Conference on Computer Communication andControl (IC4) Indore India September 2015

[9] L Boon-Leng L Dae-Seok and L Boon-Giin ldquoMobile-basedwearable-type of driver fatigue detection by GSR and EMGrdquoin Proceedings of the TENCON 2015-2015 IEEE Region 10Conference Macau China November 2015

[10] J Yan H Kuo Y Lin and T Liao ldquoReal-time driverdrowsiness detection system based on PERCLOS and gray-scale image processingrdquo in Proceedings of the 2016 Interna-tional Symposium on Computer Consumer and Control(IS3C) pp 243ndash246 Xirsquoan China July 2016

[11] G Niu and C Wang ldquoDriver fatigue features extractionrdquoMathematical Problems in Engineering vol 2014 Article ID860517 10 pages 2014

[12] L M Bergasa and J Nuevo ldquoReal-time system for monitoringdriver vigilancerdquo in Proceedings of the IEEE InternationalSymposium on Industrial Electronics 2005 ISIE 2005pp 1303ndash1308 Dubrovnik Croatia June 2005

[13] F You Y-h Li L Huang K Chen R-h Zhang and J-m XuldquoMonitoring driversrsquo sleepy status at night based on machinevisionrdquo Multimedia Tools and Applications vol 76 no 13pp 14869ndash14886 2017

[14] R-H Zhang Z-C He H-W Wang F You and K-N LildquoStudy on self-tuning tyre friction control for developingmain-servo loop integrated chassis control systemrdquo IEEEAccess vol 5 pp 6649ndash6660 2017

[15] F Chen M Song and X Ma ldquoInvestigation on the injuryseverity of drivers in rear-end collisions between cars using arandom parameters bivariate ordered probit modelrdquo Inter-national Journal of Environmental Research and PublicHealth vol 16 no 14 p 2632 2019

[16] S Yang P Luo C C Loy and X Tang ldquoWider face a facedetection benchmarkrdquo in Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR)pp 5525ndash5533 IEEE Computer Society Las Vegas NV USAJune 2016


[17] S Luo C Xu and H Li ldquoAn application of object detectionbased on YOLOv3 in trafficrdquo in Proceedings of the 2019 In-ternational Conference on Image Video and Signal Processing -IVSP 2019 pp 68ndash72 Association for Computing MachineryShanghai China 2019

[18] X Chen X Luo X Liu and J Fang ldquoEyes localization al-gorithm based on prior MTCNN face detectionrdquo in Pro-ceedings of the 2019 IEEE 8th Joint International InformationTechnology and Artificial Intelligence Conference (ITAIC)pp 1763ndash1767 Chongqing China May 2019

[19] D Sommer and M Golz ldquoEvaluation of PERCLOS basedcurrent fatigue monitoring technologiesrdquo in Proceedings ofthe 2010 Annual International Conference of the IEEE Engi-neering in Medicine and Biology pp 4456ndash4459 BuenosAires Argentina August 2010

[20] X Sun H Zhang W Meng R Zhang K Li and T PengldquoPrimary resonance analysis and vibration suppression for theharmonically excited nonlinear suspension system using apair of symmetric viscoelastic buffersrdquo Nonlinear Dynamicsvol 94 no 2 pp 1243ndash1265 2018

[21] G Wu F Chen X Pan M Xu and X Zhu ldquoUsing the visualintervention influence of pavement markings for ruttingmitigationndashpart I preliminary experiments and field testsrdquoInternational Journal of Pavement Engineering vol 20 no 6pp 734ndash746 2019

[22] P Viola and M Jones ldquoRobust real-time face detectionrdquo inProceedings Eighth IEEE International Conference on Com-puter Vision ICCV 2001 vol 2 p 747 2001

[23] K Luu C Zhu C Bhagavatula T H N Le and M SavvidesldquoA Deep learning approach to joint face detection and seg-mentationrdquo in Advances in Face Detection and Facial ImageAnalysis pp 1ndash12 Springer International Publishing ChamSwitzerland 2016

[24] J Xiang and G Zhu ldquoJoint face detection and facial ex-pression recognition with MTCNNrdquo in Proceedings of the 4thInternational Conference on Information Science and ControlEngineering pp 424ndash427 Institute of Electrical and Elec-tronics Engineers Inc Hunan China July 2017

[25] W Shi J Li and Y Yang ldquoFace fatigue detection methodbased on MTCNN and machine visionrdquo Advances in Intel-ligent Systems and Computing Springer Verlag vol 1017pp 233ndash240 Huainan China 2020

[26] S Zhao H Song W Cong Q Qi and H Tian ldquoEnd-to-endcascade cnn for simultaneously face detection and alignmentrdquoin Proceedings of the 2017 International Conference on VirtualReality and Visualization (ICVRV) pp 35ndash40 Institute ofElectrical and Electronics Engineers Inc 2017 ZhengzhouChina

[27] M El-Arabawy S Zaki and F Harby ldquoImproved AdaBoostalgorithm for face detectionrdquo in Proceedings of the 2010 In-ternational Conference on Image Processing Computer Visionand Pattern Recognition vol 1 CSREA Press Las Vegas NVUSA pp 353ndash358 2010

[28] J Redmon S Divvala R Girshick and A Farhadi ldquoYou onlylook once unified real-time object detectionrdquo in Proceedingsof the 2016 IEEE Conference on Computer Vision and PatternRecognition (CVPR) pp 779ndash788 IEEE Computer SocietyLas Vegas NV USA 2016

[29] A Krizhevsky I Sutskever and G E Hinton ldquoImageNetclassification with deep convolutional neural networksrdquoCommunications of the ACM vol 60 no 6 pp 84ndash90 2017

[30] Y Lecun L Bottou Y Bengio and P Haffner ldquoGradient-based learning applied to document recognitionrdquo Proceedingsof the IEEE vol 86 no 11 pp 2278ndash2324 1998

[31] K D E Dlib-ml ldquoA machine learning toolkitrdquo Journal ofMachine Learning Research vol 10 pp 1755ndash1758 2009

[32] J H Friedman ldquoGreedy function approximation a gradientboosting machinerdquo He Annals of Statistics vol 29 no 5pp 1189ndash1232 2001

[33] X Cao YWei FWen and J Sun ldquoFace alignment by explicitshape regressionrdquo International Journal of Computer Visionvol 107 no 2 pp 177ndash190 2014

[34] P Dollar P Welinder and P Perona ldquoCascaded pose re-gressionrdquo in Proceedings of the 2010 IEEE Computer SocietyConference on Computer Vision and Pattern RecognitionCVPR 2010 pp 1078ndash1085 IEEE Computer Society SanFrancisco CA USA June 2010

[35] H Wang F You X Chu X Li and X Sun ldquoResearch oncustomer marketing acceptance for future automatic driving-a case study in China cityrdquo IEEE Access vol 7 pp 20938ndash20949 2019

[36] L Jiang H Wang S Gao and S Jiang ldquoResearch of theautomotive driver fatigue driving early warning systemrdquoCommunications in Computer and Information ScienceSpringer-Verlag Berlin Germany pp 383ndash391 2011

[37] G Sun Y Jin Z Li F Zhang and L Jia ldquoA vision-based headstatus judging algorithm for driving fatigue detection systemrdquoAdvances in Transportation Studies vol 2015 pp 51ndash64 2015

[38] C E Shannon ldquoA mathematical theory of communicationrdquoBell System Technical Journal vol 27 1948

[39] S Abtahi M Omidyeganeh S Shirmohammadi andB Hariri ldquoYawDDrdquo in Proceedings of the 5th ACM Multi-media Systems Conference MMSys 2014 pp 24ndash28 Associ-ation for Computing Machinery Singapore March 2014

[40] Z You Y Gao J Zhang H Zhang M Zhou and C Wu ldquoAstudy on driver fatigue recognition based on SVMmethodrdquo inProceedings of the 4th International Conference on Trans-portation Information and Safety ICTIS 2017 pp 693ndash697Institute of Electrical and Electronics Engineers Inc BanffCanada August 2017

[41] J Hernandez-Orallo ldquoROC curves for regressionrdquo PatternRecognition vol 46 no 12 pp 3395ndash3411 2013

[42] L Tychsen-Smith and L Petersson ldquoImproving object lo-calization with fitness NMS and bounded IoU lossrdquo in Pro-ceedings of the 31st Meeting of the IEEECVF Conference onComputer Vision and Pattern Recognition CVPR 2018pp 6877ndash6885 IEEE Computer Society Salt Lake City UTUSA June 2018

[43] G Lei X Liang Z Xiao and Y Li ldquoReal-time driver fatiguedetection based on morphology infrared features and deeplearningrdquo Infrared amp Laser Engineering vol 47 no 2 ArticleID 203009 2018

[44] J M Guo and M Herleeyandi ldquoDriver drowsiness detectionusing hybrid convolutional neural network and long short-term memoryrdquo Multimedia Tools amp Applications vol 78no 20 pp 29059ndash29087 2019

[45] C Xu Y Yang S Jin Z Qu and L Hou ldquoPotential risk andits influencing factors for separated bicycle pathsrdquo AccidentAnalysis amp Prevention vol 87 pp 59ndash67 2016

[46] F Chen H Peng X Ma J Liang W Hao and X PanldquoExamining the safety of trucks under crosswind at bridge-tunnel section a driving simulator studyrdquo Tunnelling andUnderground Space Technology vol 92 Article ID 1030342019

[47] H Xiong X Zhu and R Zhang ldquoEnergy recovery strategynumerical simulation for dual axle drive pure electric vehiclebased on motor loss model and big data calculationrdquo Com-plexity vol 2018 Article ID 4071743 14 pages 2018


[48] X Qu M Zhou Y Yu C T Lin and X Wang ldquoJointlydampening traffic oscillations and improving energy con-sumption with electric connected and automated vehicles areinforcement learning based approachrdquo Applied Energyvol 257 Article ID 114030 2019

[49] M Zhou Y Yu and X Qu ldquoDevelopment of an efficientdriving strategy for connected and automated vehicles atsignalized intersections a reinforcement learning approachrdquoIEEE Transactions on Intelligent Transportation Systemsvol 21 no 1 pp 433ndash443 2019


12 Detection Methods Based on Machine Vision Withdistinctive characteristics of the vehicle motion and thebehaviors of the driver obtained this method assesses thedriverrsquos fatigue status Machine vision-based detection hasbecome the widely used method in fatigue driving detectiondue to its noninvasion and higher accuracy is methodapplies core technologies including face detection eye po-sitioning and fatigue assessment Yan et al [10] used themask to locate the eye position by obtaining the driverrsquosfacial image and used PERCLOS to evaluate the driverrsquosfatigue state is method has better performance on in-dividuals with conspicuous features but the fabrication ofthe mask has a significant influence on the generalizationperformance of the model Niu and Wang [11] divided theface image in the sequence image into nonoverlappingblocks of the same size en they managed to use Gaborwavelet transform to extract multiscale features In order toselect the most recognizable ones they applied AdaBoostalgorithm is method can effectively recognize differentgenders and postures under various illumination conditionsUsing ldquobright eye effectrdquo Bergasa and Nuevo [12] located eyepositionwith active near-infrared light source equipmenteyused finite-state machine to confirm whether the eye is closedey also applied fuzzy system to evaluate the fatigue stateHowever Bergasarsquos algorithm depends highly on hardwarelevel on the other hand the effectiveness of the ldquobright eyeeffectrdquo strictly relies on surrounding light conditions You et al[13] applied the CAMShift tracking algorithm to make thetargeted areas detectable even they were under occlusionenthe eye feature points were obtained according to the specificproportion relationship of the facial organs Finally they usedPERCLOS to determine driver fatigue state

13 Detection Methods Based on Information Fusion Anyfatigue detection method has its advantages and disadvan-tages So comprehensive monitoring of driver fatigue statusby various methods is promising ldquoAWAKErdquo [14] launchedby the European Union is a driving behavior comprehensivemonitoring system It used many sensors such as images andpressures to synthesize the driverrsquos eye movement the di-rection of eyesight steering wheel grip and other drivingconditions en it made comprehensive detection andevaluation Seeing Machines [15] conducts multifeatureinformation fusion by detecting facial features such asdriverrsquos head posture eyelid movement gaze direction andpupil diameter It completed real-time monitoring of driverfatigue status

Although the technology of fatigue detection has madegreat progress it can be better

(i) Physiology-based driver fatigue detections require avariety of additional monitoring devices or equip-ment It would not only reduce comfort duringdriving but also make the collected data costly andvulnerable which has set back the popularization ofthese methods

(ii) If the light condition changes or the driverrsquos face ispartially occluded for example wearing glasses or

sunglasses AdaBoost fails to accurately locate theface position and give the alarm to the driverpromptly

(iii) At present the commonly used algorithms arebased on PERCLOS which judge fatigue by openingand closing state of the driverrsquos eyes However whenthe driverrsquos eyes are too small the algorithms areeasy to misjudge Moreover other fatigue indicatorsare less commonly used due to lower reliability andless robustness

As above literature studies discussed results of thedriving fatigue detection have defects of high intrusion lowrobustness and low reliability erefore we propose afatigue driving detection algorithm based on facial motioninformation entropy e innovations are as follows

(i) We design a driverrsquos face detection architecturebased on the improved YOLOv3-tiny convolutionalneural network and train the network with theopen-source data setWIDER FACE [16] Comparedwith other deep learning algorithms such asYOLOv3 [17] and MTCNN [18] the algorithmbased on the improved YOLOv3-tiny network ismore accurate and simplified It has fewer calcu-lations and thus is easy to transplant to othermobiles

(ii) We used the Dlib toolkit to extract facial featurepoints recognized by improved YOLOv3-tinyconvolutional neural network en we created theFFTafter analyzing the characteristics of the eye andmouth position Next we constructed FFV whichcontains the overall information of the area andcentroid of each FFT We calculate the FFV of eachframe and write it to the database ereby a stateanalysis data set is established In many researchstudies the basis for assessing the state of the driveris the recognition result of a single frame or a fewframes which reduce the accuracy of fatigue drivingdetection Based on the analysis results of a largenumber of consecutive frames we design slidingwindows of driving fatigue analysis to obtain thestatistical characteristics of the facial motion stateerefore the process of driver fatigue can beobserved

(iii) To get rid of the interference that originated fromthe size differences between every FFT we intro-duce the face projection datum plane and apply theprojection principle to extract the motion featurepoints of the faceen based on the motion featurepoints we propose the facial motion informationentropy which quantitatively characterizes thechaotic degree of the motion feature points of theface Accordingly the driverrsquos fatigue state can bejudged At present the commonly used algorithmsare based on PERCLOS [19] which judge fatigue byopening and closing state of the driverrsquos eyesHowever when the driverrsquos eyes are too small thealgorithms are easy to misjudge erefore we




2 Methodology




















Driving video

ImprovedYOLOv3-tiny


location

Face

pos

ition

ing


windows

Fatig

ue ju

dgm

ent

FFV

Feat

ure v

ecto

r ext

ract

ion

Dat

a set

bui

ldin

g

Fatiguejudgment

H_FFFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV


10

11


















DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL DBL DBL Conv


Up

Sam

ple

LeakyReLU

BatchNormConv









1113954S(t+1)

1113954S(t)

+ Rt h I 1113954S(t)

1113874 11138751113874 1113875 t 1 T (2)









(a) (b) (c) (d) (e) (f )



FFV Fx FyS

radic1113872 1113873 (3)


Fx Ax + Bx + Cx

3

Fy Ay + By + Cy

3


11138681113868111386811138681113868

11138681113868111386811138681113868

2

(4)


Ax Ay1113872 1113873 p36x + p39x

2p36y + p39y

21113888 1113889

Bx By1113872 1113873 p42x + p45x

2p42y + p45y

21113888 1113889

Cx Cy1113872 1113873 p60x + p64x

2p60y + p64y

21113888 1113889

(5)




0

1

2

3

4

5

6

7 8 9

10

11

12

13

14

15

16

1718 19 20

21 2223 24 25

26

27

28

29

3031 3233 34 35

36 37 38394041 42

43 44454647

48 49 50 51 52 5354

55565758

5960

61 62 63 64656667

(a) (b)


A B

C




x Fx minuscol2

1113888 1113889lowast

S

S0

1113971

+col2

y Fy minusrow2

1113874 1113875lowast

S

S0

1113971

+row2

(6)




210020001900180017001600150014001300

270260

250240

230220

210200

50 100 150 200300250

350X

Y

Z

LeftNormalRight

(a) (b) (c)




S2

S0

S1







p(X) logp(X) (7)



Fx ΣFx

N

Fy ΣFy

N

(8)


li


+ Fy minus Fy1113872 11138732

1113970

(9)


μl 1113936

Ni1 li

N

σl

1113936Ni1 li minus μl( 1113857

2

N

1113971

(10)



σl

ilowastμl

σl

1113890 1113891 (11)


μlσl

+ 1 (12)



HF(X) minus 1113944

imax

i1p xi( 1113857 logp xi( 1113857 p xi( 1113857

ni

N (13)





wlowastT

middot x + blowast

0 (14)

195 210 220200 205 225215190X

2000

2025

2050

2075

2100

2125

2150

2175

Y




minwb

J(w) 12w

22


⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(15)









2000

2025

2050

2075

2100

2125

2150

2175Y

195 215210 220190 200 225205X

(a)

195 215210 220190 200 225205X

2000

2025

2050

2075

2100

2125

2150

2175

Y

(b)







FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV



Start

Video stream


Facedetected



N

Y Y N

N

Y

N Y

Next frame

Calculate FFV


FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N












ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References
























































2 Methodology




















Driving video

ImprovedYOLOv3-tiny


location

Face

pos

ition

ing


windows

Fatig

ue ju

dgm

ent

FFV

Feat

ure v

ecto

r ext

ract

ion

Dat

a set

bui

ldin

g

Fatiguejudgment

H_FFFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV


10

11


















DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL DBL DBL Conv


Up

Sam

ple

LeakyReLU

BatchNormConv









1113954S(t+1)

1113954S(t)

+ Rt h I 1113954S(t)

1113874 11138751113874 1113875 t 1 T (2)









(a) (b) (c) (d) (e) (f )



FFV Fx FyS

radic1113872 1113873 (3)


Fx Ax + Bx + Cx

3

Fy Ay + By + Cy

3


11138681113868111386811138681113868

11138681113868111386811138681113868

2

(4)


Ax Ay1113872 1113873 p36x + p39x

2p36y + p39y

21113888 1113889

Bx By1113872 1113873 p42x + p45x

2p42y + p45y

21113888 1113889

Cx Cy1113872 1113873 p60x + p64x

2p60y + p64y

21113888 1113889

(5)




0

1

2

3

4

5

6

7 8 9

10

11

12

13

14

15

16

1718 19 20

21 2223 24 25

26

27

28

29

3031 3233 34 35

36 37 38394041 42

43 44454647

48 49 50 51 52 5354

55565758

5960

61 62 63 64656667

(a) (b)


A B

C




x Fx minuscol2

1113888 1113889lowast

S

S0

1113971

+col2

y Fy minusrow2

1113874 1113875lowast

S

S0

1113971

+row2

(6)




210020001900180017001600150014001300

270260

250240

230220

210200

50 100 150 200300250

350X

Y

Z

LeftNormalRight

(a) (b) (c)




S2

S0

S1







p(X) logp(X) (7)



Fx ΣFx

N

Fy ΣFy

N

(8)


li


+ Fy minus Fy1113872 11138732

1113970

(9)


μl 1113936

Ni1 li

N

σl

1113936Ni1 li minus μl( 1113857

2

N

1113971

(10)



σl

ilowastμl

σl

1113890 1113891 (11)


μlσl

+ 1 (12)



HF(X) minus 1113944

imax

i1p xi( 1113857 logp xi( 1113857 p xi( 1113857

ni

N (13)





wlowastT

middot x + blowast

0 (14)

195 210 220200 205 225215190X

2000

2025

2050

2075

2100

2125

2150

2175

Y




minwb

J(w) 12w

22


⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(15)









2000

2025

2050

2075

2100

2125

2150

2175Y

195 215210 220190 200 225205X

(a)

195 215210 220190 200 225205X

2000

2025

2050

2075

2100

2125

2150

2175

Y

(b)







FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV



Start

Video stream


Facedetected



N

Y Y N

N

Y

N Y

Next frame

Calculate FFV


FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N












ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References

































































Driving video

ImprovedYOLOv3-tiny


location

Face

pos

ition

ing


windows

Fatig

ue ju

dgm

ent

FFV

Feat

ure v

ecto

r ext

ract

ion

Dat

a set

bui

ldin

g

Fatiguejudgment

H_FFFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV


10

11


















DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL

Max

Poo

l

DBL DBL DBL Conv


Up

Sam

ple

LeakyReLU

BatchNormConv









1113954S(t+1)

1113954S(t)

+ Rt h I 1113954S(t)

1113874 11138751113874 1113875 t 1 T (2)









(a) (b) (c) (d) (e) (f )



FFV Fx FyS

radic1113872 1113873 (3)


Fx Ax + Bx + Cx

3

Fy Ay + By + Cy

3


11138681113868111386811138681113868

11138681113868111386811138681113868

2

(4)


Ax Ay1113872 1113873 p36x + p39x

2p36y + p39y

21113888 1113889

Bx By1113872 1113873 p42x + p45x

2p42y + p45y

21113888 1113889

Cx Cy1113872 1113873 p60x + p64x

2p60y + p64y

21113888 1113889

(5)




0

1

2

3

4

5

6

7 8 9

10

11

12

13

14

15

16

1718 19 20

21 2223 24 25

26

27

28

29

3031 3233 34 35

36 37 38394041 42

43 44454647

48 49 50 51 52 5354

55565758

5960

61 62 63 64656667

(a) (b)


A B

C




x Fx minuscol2

1113888 1113889lowast

S

S0

1113971

+col2

y Fy minusrow2

1113874 1113875lowast

S

S0

1113971

+row2

(6)




210020001900180017001600150014001300

270260

250240

230220

210200

50 100 150 200300250

350X

Y

Z

LeftNormalRight

(a) (b) (c)




S2

S0

S1







p(X) logp(X) (7)



Fx ΣFx

N

Fy ΣFy

N

(8)


li


+ Fy minus Fy1113872 11138732

1113970

(9)


μl 1113936

Ni1 li

N

σl

1113936Ni1 li minus μl( 1113857

2

N

1113971

(10)



σl

ilowastμl

σl

1113890 1113891 (11)


μlσl

+ 1 (12)



HF(X) minus 1113944

imax

i1p xi( 1113857 logp xi( 1113857 p xi( 1113857

ni

N (13)





wlowastT

middot x + blowast

0 (14)

195 210 220200 205 225215190X

2000

2025

2050

2075

2100

2125

2150

2175

Y




minwb

J(w) 12w

22


⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(15)









2000

2025

2050

2075

2100

2125

2150

2175Y

195 215210 220190 200 225205X

(a)

195 215210 220190 200 225205X

2000

2025

2050

2075

2100

2125

2150

2175

Y

(b)







FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV



Start

Video stream


Facedetected



N

Y Y N

N

Y

N Y

Next frame

Calculate FFV


FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N












ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References



























































1113954S(t+1)

1113954S(t)

+ Rt h I 1113954S(t)

1113874 11138751113874 1113875 t 1 T (2)









(a) (b) (c) (d) (e) (f )



FFV Fx FyS

radic1113872 1113873 (3)


Fx Ax + Bx + Cx

3

Fy Ay + By + Cy

3


11138681113868111386811138681113868

11138681113868111386811138681113868

2

(4)


Ax Ay1113872 1113873 p36x + p39x

2p36y + p39y

21113888 1113889

Bx By1113872 1113873 p42x + p45x

2p42y + p45y

21113888 1113889

Cx Cy1113872 1113873 p60x + p64x

2p60y + p64y

21113888 1113889

(5)




0

1

2

3

4

5

6

7 8 9

10

11

12

13

14

15

16

1718 19 20

21 2223 24 25

26

27

28

29

3031 3233 34 35

36 37 38394041 42

43 44454647

48 49 50 51 52 5354

55565758

5960

61 62 63 64656667

(a) (b)


A B

C




x Fx minuscol2

1113888 1113889lowast

S

S0

1113971

+col2

y Fy minusrow2

1113874 1113875lowast

S

S0

1113971

+row2

(6)




210020001900180017001600150014001300

270260

250240

230220

210200

50 100 150 200300250

350X

Y

Z

LeftNormalRight

(a) (b) (c)




S2

S0

S1







p(X) logp(X) (7)



Fx ΣFx

N

Fy ΣFy

N

(8)


li


+ Fy minus Fy1113872 11138732

1113970

(9)


μl 1113936

Ni1 li

N

σl

1113936Ni1 li minus μl( 1113857

2

N

1113971

(10)



σl

ilowastμl

σl

1113890 1113891 (11)


μlσl

+ 1 (12)



HF(X) minus 1113944

imax

i1p xi( 1113857 logp xi( 1113857 p xi( 1113857

ni

N (13)





wlowastT

middot x + blowast

0 (14)

195 210 220200 205 225215190X

2000

2025

2050

2075

2100

2125

2150

2175

Y




minwb

J(w) 12w

22


⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(15)









2000

2025

2050

2075

2100

2125

2150

2175Y

195 215210 220190 200 225205X

(a)

195 215210 220190 200 225205X

2000

2025

2050

2075

2100

2125

2150

2175

Y

(b)







FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV



Start

Video stream


Facedetected



N

Y Y N

N

Y

N Y

Next frame

Calculate FFV


FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N












ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References






















































FFV Fx FyS

radic1113872 1113873 (3)


Fx Ax + Bx + Cx

3

Fy Ay + By + Cy

3


11138681113868111386811138681113868

11138681113868111386811138681113868

2

(4)


Ax Ay1113872 1113873 p36x + p39x

2p36y + p39y

21113888 1113889

Bx By1113872 1113873 p42x + p45x

2p42y + p45y

21113888 1113889

Cx Cy1113872 1113873 p60x + p64x

2p60y + p64y

21113888 1113889

(5)




0

1

2

3

4

5

6

7 8 9

10

11

12

13

14

15

16

1718 19 20

21 2223 24 25

26

27

28

29

3031 3233 34 35

36 37 38394041 42

43 44454647

48 49 50 51 52 5354

55565758

5960

61 62 63 64656667

(a) (b)


A B

C




x Fx minuscol2

1113888 1113889lowast

S

S0

1113971

+col2

y Fy minusrow2

1113874 1113875lowast

S

S0

1113971

+row2

(6)




210020001900180017001600150014001300

270260

250240

230220

210200

50 100 150 200300250

350X

Y

Z

LeftNormalRight

(a) (b) (c)




S2

S0

S1







p(X) logp(X) (7)



Fx ΣFx

N

Fy ΣFy

N

(8)


li


+ Fy minus Fy1113872 11138732

1113970

(9)


μl 1113936

Ni1 li

N

σl

1113936Ni1 li minus μl( 1113857

2

N

1113971

(10)



σl

ilowastμl

σl

1113890 1113891 (11)


μlσl

+ 1 (12)



HF(X) minus 1113944

imax

i1p xi( 1113857 logp xi( 1113857 p xi( 1113857

ni

N (13)





wlowastT

middot x + blowast

0 (14)

195 210 220200 205 225215190X

2000

2025

2050

2075

2100

2125

2150

2175

Y




minwb

J(w) 12w

22


⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(15)









2000

2025

2050

2075

2100

2125

2150

2175Y

195 215210 220190 200 225205X

(a)

195 215210 220190 200 225205X

2000

2025

2050

2075

2100

2125

2150

2175

Y

(b)







FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV



Start

Video stream


Facedetected



N

Y Y N

N

Y

N Y

Next frame

Calculate FFV


FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N












ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References























































x Fx minuscol2

1113888 1113889lowast

S

S0

1113971

+col2

y Fy minusrow2

1113874 1113875lowast

S

S0

1113971

+row2

(6)




210020001900180017001600150014001300

270260

250240

230220

210200

50 100 150 200300250

350X

Y

Z

LeftNormalRight

(a) (b) (c)




S2

S0

S1







p(X) logp(X) (7)



Fx ΣFx

N

Fy ΣFy

N

(8)


li


+ Fy minus Fy1113872 11138732

1113970

(9)


μl 1113936

Ni1 li

N

σl

1113936Ni1 li minus μl( 1113857

2

N

1113971

(10)



σl

ilowastμl

σl

1113890 1113891 (11)


μlσl

+ 1 (12)



HF(X) minus 1113944

imax

i1p xi( 1113857 logp xi( 1113857 p xi( 1113857

ni

N (13)





wlowastT

middot x + blowast

0 (14)

195 210 220200 205 225215190X

2000

2025

2050

2075

2100

2125

2150

2175

Y




minwb

J(w) 12w

22


⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(15)









2000

2025

2050

2075

2100

2125

2150

2175Y

195 215210 220190 200 225205X

(a)

195 215210 220190 200 225205X

2000

2025

2050

2075

2100

2125

2150

2175

Y

(b)







FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV



Start

Video stream


Facedetected



N

Y Y N

N

Y

N Y

Next frame

Calculate FFV


FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N












ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References


























































p(X) logp(X) (7)



Fx ΣFx

N

Fy ΣFy

N

(8)


li


+ Fy minus Fy1113872 11138732

1113970

(9)


μl 1113936

Ni1 li

N

σl

1113936Ni1 li minus μl( 1113857

2

N

1113971

(10)



σl

ilowastμl

σl

1113890 1113891 (11)


μlσl

+ 1 (12)



HF(X) minus 1113944

imax

i1p xi( 1113857 logp xi( 1113857 p xi( 1113857

ni

N (13)





wlowastT

middot x + blowast

0 (14)

195 210 220200 205 225215190X

2000

2025

2050

2075

2100

2125

2150

2175

Y




minwb

J(w) 12w

22


⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(15)









2000

2025

2050

2075

2100

2125

2150

2175Y

195 215210 220190 200 225205X

(a)

195 215210 220190 200 225205X

2000

2025

2050

2075

2100

2125

2150

2175

Y

(b)







FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV



Start

Video stream


Facedetected



N

Y Y N

N

Y

N Y

Next frame

Calculate FFV


FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N












ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References























































minwb

J(w) 12w

22


⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(15)









2000

2025

2050

2075

2100

2125

2150

2175Y

195 215210 220190 200 225205X

(a)

195 215210 220190 200 225205X

2000

2025

2050

2075

2100

2125

2150

2175

Y

(b)







FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV



Start

Video stream


Facedetected



N

Y Y N

N

Y

N Y

Next frame

Calculate FFV


FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N












ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References


























































FFV data setSliding

windows

FFV

FFV

FFV

FFV

FFV

FFV

FFV

FFV



Start

Video stream


Facedetected



N

Y Y N

N

Y

N Y

Next frame

Calculate FFV


FFV data set

Calculate H_F

H_F lt 132

Fatigue Nonfatigue

Data settraversed

System quit

Y

N












ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References



























































ACR Ndetected

Ntotal (16)




(17)



Eye open Fps 248


(a)

Eye open Fps 278


(b)

Eye open Fps 249

Face yes Mouth big

(c)




Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References























































Sn TP

TP + FN

Sp TN

TN + FP

(18)

(a) (b) (c) (d)

(e) (f ) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)


Face_d cap face

Face_d

Face











0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References






























































0

1000

0

2000

0

3000

0

4000

0

5000

0

6000

0

7000

0

8000

0

9000

0

1000

00

Steps


10

09

08

07

06

05

04

03

02

01

00

ACR

0985


ROCRandom chance

08 10402 0601 ndash Sp

0

02

04

06

08

1S n

Figure 16 ROC curve






















Data Availability




Acknowledgments


References










































































Data Availability




Acknowledgments


References






















































afatiguedrivingdetectionalgorithmbasedonfacialmotion...

Documents