Understanding Head and Hand Activities and Coordination inNaturalistic Driving Videos

Sujitha Martin1, Eshed Ohn-Bar1, Ashish Tawari1 and Mohan M. Trivedi1

Abstract— In this work, we propose a vision-based analysisframework for recognizing in-vehicle activities such as interac-tions with the steering wheel, the instrument cluster and thegear. The framework leverages two views for activity analysis,a camera looking at the driver’s hand and another looking atthe driver’s head. The techniques proposed can be used byresearchers in order to extract ‘mid-level’ information fromvideo, which is information that represents some semanticunderstanding of the scene but may still require an expertin order to distinguish difficult cases or leverage the cues toperform drive analysis. Unlike such information, ’low-level’video is large in quantity and can’t be used unless processedentirely by an expert. This work can apply to minimizingmanual labor so that researchers may better benefit from theaccessibility of the data and provide them with the ability toperform larger-scaled studies.

I. INTRODUCTION

For the past 50 years, most of the data related to ve-hicular collisions have come from post-crash analysis. Onlyrecently, naturalistic driving studies (NDS) began providingdetailed information about driver behavior, vehicle state, androadways using video cameras and other types of sensors.Consequently, such data holds the key for the role and effectof cognitive processes, in-vehicle dynamics, and surroundingsalient objects on driver behavior [1], [2].

The 100-Car Naturalistic Driving Study is the firstinstrumented-vehicle study undertaken with the primary pur-pose of collecting large-scale, naturalistic driving data. A2006 report on the results of the 100-car field experiment [3]revealed that almost 80 percent of all crashes and 65 percentof all near-crashes involved the driver looking away fromthe forward roadway just prior to the onset of the conflict. Itwas also shown that 67% of crashes and 82% of near-crashesoccurred when subject vehicle drivers were driving with atleast one hand on the wheel. More details about the presenceor absence of driver’s hands on the wheel and the driver’sinattention to forward roadway, for crashes and near-crashesas reported in [3] are shown in Table I and Table II.

Because of the above issues, on-road analysis of driverbehavior is becoming an increasingly essential componentfor future advanced driver assistance system [4]. Towardsthis end, we focus on analyzing where and what the driver’shands do in the vehicle. Hand positions can provide the levelof control drivers exhibit during a maneuver or can even givesome information about mental workload [5]. Furthermore,

1The authors are with the Laboratory of Intelligent andSafe Automobiles at the University of California, San Diego,USA scmartin@ucsd.edu,eohnbar@ucsd.edu,atawari@ucsd.edu, mtrivedi@ucsd.edu

in-vehicle activities involving hand movements often demandcoordination with head and eye movements. For this, a dis-tributed camera setup is installed to simultaneously observehand and head movements. Together, this multiperspectiveapproach allows us to derive a semantic level representationof driver activities, similar to research studies on upper bodybased gesture analysis for intelligent vehicles [6] and smartenvironments [7].

Hands on wheel Crash (%) Near-Crash (%)Left hand only 30.4 31.7

Unknown 29.0 15.1Both hands 24.6 35.1

Right hand only 11.6 15.5No hands on wheel 4.4 2.6

TABLE I: Hands on wheel when crash and near-crashoccurred from 100-car study [3]

Inattention to for-ward roadway

Crash (%) Near-Crash(%)

Left window 9.7 3.2Talking/listening 8.3 4.8Passenger in adjacentseat

6.9 6.1

Center mirror 1.4 1.8Right window 1.4 1.8In-vehicle controls -Other

1.4 0.0

Adjust radio 0.0 1.3

TABLE II: Inattention to forward roadway when crash andnear-crash occurred from 100-car study [3]

The approach is purely vision-based, with no markers orintrusive devices. There are several challenges that such asystem must overcome, both for the robust extraction of head[8] and hand cues [9]. For the head, there are challengesof self-occlusion due to large head motion and of privacyimplication for drivers in large scale data. Interestingly, arecent study has focused on the design of deidentificationfilters to protect the privacy of drivers while preservingdriver behavior [10]. For the hand, detection is challengingas the human hand is highly deformable and tends to occludeitself in images. The problem is further complicated by thevehicular requirement for algorithms to be robust to changing

IEEE Intelligent Vehicles Symposium 2014

Probability Output

𝑝1(𝑡) ⋮

𝑝4(𝑡)

𝑝1(𝑡)⋮

𝑝4(𝑡)𝜙(𝑡)

Offline Discriminative

Classifier Training

Video

Regions of Interest Extraction Feature Extraction

Model 1 𝑝1(𝑡)

(Wheel) ⋮

Model 4 𝑝4(𝑡)

(Instruments) Semantic

Classification

Co

lor

Dep

th

Edge, Color, Texture Cues Hand Cues

Face Detection

and Tracking

Pose and Landmark

Cues 𝝓(𝑡) Head Cues

Co

lor

Dep

th

𝒑 𝑡

𝝓 𝑡

Integrated Cues

Analysis

Fig. 1: The proposed approach for driver activity recognition. Head and hand cues are extracted from color and depth videoin regions of interest. A classifier provides an integration of the cues, and the final activity classification.

illumination. Therefore, we are interested in incorporatinghead and eye cues to better represent the driver’s interactionwith the steering wheel, the instrument cluster and the gear.A more detailed analysis of various feature extraction onindividual perspectives and their integration can be found in[11].

II. ACTIVITY ANALYSIS FRAMEWORK

The framework in this work leverages two views foractivity analysis, a camera looking at the driver’s hand andanother looking at the head. As shown in Fig. 1, these areintegrated in order to produce the final activity classification.

A. Hand Cues

Localizing the hands in the vehicle with a high degree ofaccuracy is highly desired. One approach for hand detectionrelies on a sliding-window. This is a common technique forgeneric visual object detection, where a model is learnedbased on positive samples (i.e. hands in different poses)of fixed size and negative samples which don’t contain theobject of interest. A classifier is then used to learn a classifi-cation rule. Such a scheme can be applied on multiple scalesof the image in order to detect objects at different sizes.Specifically for hand detection, these techniques are facedwith challenges as the hand is highly deformable and tendsto occlude itself. Models are often sensitive to even small in-plane rotation [12] and deformation. A more sophisticated setof models (usually referred to in literature as a ‘part-baseddeformable model’ [13]) allows for learning a model fordifferent configurations, deformations, and occlusion types.A pre-trained model for hand shape, however, resulted inmany false positives on naturalistic driving dataset [14], [15].

Instead of learning a model for hand and searching forit throughout the entire cabin, we constrain the problem to

a number of regions of interest which may be useful forstudying the driver’s state. This provides several benefits:

1) As the variation in hand appearance differs based onthe region in which it is in, a model learned foreach region could potentially better generalize over thevariations in that specific region.

2) This phrasing of the problem allows us to study theperformance of visual descriptors for each region. Forinstance, some regions are less prone to illuminationchanges.

3) Integration: in the context of our problem, the handmay be commonly found in only parts of the scene.Assuming that the hands must be in one of three re-gions of interest reduces the complexity of the problemand opens up the door for leveraging cues among thedifferent regions. Integration also provides a modelwith the opportunity to perform higher-level reasoningof the hands configuration.

Our approach attempts to separate the scene into differ-ently sized regions, and model two classes: no hand and handpresence. To that end, a linear kernel binary support vectormachine (SVM) classifier is trained where input featuresare Histogram of Orientations (HOG) as applied in multiplescales. The linear SVM is used to learn a hand presencemodel in each of the periphery regions (the side hand rest,gear shift, and instrument cluster) and a ‘two hands on thewheel’ model for the wheel region. LIBSVM [16] allows forapproximating the probability for hand presence in each ofthe regions at time t,

p(t) =

p1(t)...pn(t)

(1)

IEEE Intelligent Vehicles Symposium 2014

Time −−−−−−−−−−→

Fig. 2: Hand, head, and eye cues can be used in order to analyze driver activity. Notice the guiding head movementsperformed in order to gather visual information before and while the hand interaction occurs.

where n is the number of regions considered. For headand hand integration, it will be useful for us to study n =3, where the three regions are the wheel, gear shift, andinstrument cluster. These probabilities are a powerful toolfor analyzing semantic information in the scene, as they eachcorrespond to our belief of a certain hand configuration. Theprobability output may be more reliable in certain regions,such as in the gear shift region, or noisier in others, suchas in the difficult wheel region which is large and prone tovolatile illumination. This motivates their integration, whichcan be done in multiple ways. A simple way which showedgood results and opens up the door for integration with otherviews and modalities (for instance, head or CAN cues) is byletting a second-stage classifier reason over the probabilitiesoutputted by the regional models. Therefore, a linear SVMis provided with the probability vector, p(t) to solve themulticlass problem and assign each frame with an activitylabel, from 1 to n.

B. Head Cues

One type of features representative of the driver’s headis head pose. Head pose estimator, however, needs to sat-isfy certain specifications to function robustly in a volatiledriving environment. Continuous Head Movement Estimator(CoHMEt) [17] outlines these necessary specifications as:automatic, real-time, wide operational range, lighting invari-ant, person invariant and occlusion tolerant. Facial features-based approaches for extracting the head pose, such as themixture of tree structure [18] and supervised descent methodfor face alignment [19], show promise of meeting manyof the requirements. An additional benefit of using facialfeatures for estimating head pose is that it allows for faciallandmark analysis, such as level of eye opening. While thepercent of eye opening has been vastly studied for detectingdriver fatigue, measuring the openness of eyes can benefit inestimating the driver’s gaze. For instance, when interactingwith the instrument panel, distinctive eye cues arise (see Fig.3). In this work, we explore the possibility of using headpose and eye opening as features in monitoring the in-vehicledriver activities, summarized in a feature vector we call asφφφ(t) at time t.

Driver interaction with the infotainment system and the

gear show unique pattern combination with head pose, eyeopening and hand locations as shown in Fig. 2. Figure 3shows time synchronized plots of head pose, eye opening,hand activity for two typical events: interacting with IP andinteracting with gear. In Fig. 3 head pose in yaw and pitchare measured in degrees, where a decreasing value in yawrepresents the driver looking rightward and an increasingvalue in pitch represents the driver looking downward. Inthe plot for eye opening, a value of 1 represents the normalsize of eyes, values greater than one could represent lookingupward, and values less than one could represent lookingdownward. Hand locations in the image plane are also plottedin a time-synchronized manner, but instead the presencesof hands in discrete locations are plotted. The green dottedline indicates the start of supportive head and eye cues tothe respective hand activity. The dotted red lines indicatesthe start and end of the presence of hand in locationsrespective of its activity. These plots show the presence ofhand, head and eye movements while the driver interactswith the infotainment system (Fig. 3(a)) and with the gear(Fig. 3(b)). While the latency of each cue is circumstantial,we experimentally validate the use of head and eye cues tostrengthen the detection of hand activity recognition.

C. Integration of Modalities and Perspectives

We obtain an SVM model trained on RGB descriptors ofeither: 1) Hand or no hand in the ROI (in the peripheralROIs) 2) Two hands or one or no hands in the ROI (thecenter wheel ROI). The assumption that the hand can onlybe found in a subset of the regions of interest allows thesecond-stage classifier to reason over the likelihood of thedriver’s two hand configuration. For instance, if the smaller,peripheral regions are known to be more reliable, and allshow a ‘no hand’ event, we would like a model that canreason in such case that both hands are on the wheel.

In addition, the second-stage classifier provides an op-portunity for integration with other modalities. Since weobserved a correlation between head dynamics and handactivity, we perform a study of head and hand cue integration.Ideally, the second-stage classifier will resolve false positivesand increase the likelihood of certain hand configurations byleveraging features extracted from the pose of the head and

IEEE Intelligent Vehicles Symposium 2014

3513 3533 3553 3573 3593−50

0

50

FrameONumber

Yaw

OLde

gree

so

3513 3533 3553 3573 3593

−40

−20

0

20

FrameONumber

Pitc

hOLd

egre

eso

3513 3533 3553 3573 35930

1

2

3

FrameONumber

Eye

OOpe

ning

3513 3533 3553 3573 3593

OtherWheel

IPGear

FrameONumber

Han

dOLo

catio

n

830 880 930 987−50

0

50

FrameyNumber

Yaw

yGde

gree

sH

830 880 930 987

−40

−20

0

20

FrameyNumber

Pitc

hyGd

egre

esH

830 880 930 9870

1

2

3

FrameyNumber

Eye

yOpe

ning

830 850 870 890 910 930 951 976 997

OtherWheel

IPGear

FrameyNumberH

andy

Loca

tion

(a) Instrument Panel Interaction (b) Gear Shift Interaction

Fig. 3: Hand, head, and eye cue visualization for (a) an instrument panel activity sequence and (b) gear shift activity sequence.Green line: indication of start of head and eye cues (yaw, pitch, and opening) before the hand activity. Red lines: start andend of the hand activity. See Section II-B for further detail on the cues.

Loction Activity TypesOn/Off RadioChange Preset

Radio Navigate to Radio ChannelIncrease/Decrease VolumeSeek/Scan for Preferred ChannelInsert/Eject CD

On/Off Hazard Lights

On/Off ACClimate Control Adjust AC

Change Fan Direction

Side Rest Adjust Mirrors

Gear Park/Exit Parking

TABLE III: Types of activities in the dataset collected.

eyes. The final feature vector is therefore denoted by

xxx(t) =

[p(t)φφφ(t)

](2)

where φφφ(t) is the features extracted from the head view.We compare two possible choices for φφφ(t). First, a simple

concatenation of the values from pose and landmarks over atime window is used. Second, we use summarizing statisticsover the time window, namely the mean, minimum, andmaximum for each of the features over the temporal window.

III. EXPERIMENTAL EVALUATION AND DISCUSSION

Detecting the driver’s activity (e.g. adjusting radio, usinggear) is an important step towards detecting driver distrac-tion. In this section, we describe the dataset and the results ofthe proposed framework. By integrating head and hand cues,we show promising results of driver’s activity recognition.

A. Experimental Setup

In order to train and test the activity recognition frame-work, we collected a dataset using two Kinects, one ob-serving the hands and one observing the head. The datasetwas collected while driving, where subjects were asked toperform tasks as listed in Table III. The four subjects (threemales and one female) were of various nationalities andranged from 20 to 30 years of age. The amount of drivingexperience varied as well, ranging from a few years to morethan a decade. The tasks in Table III were first practicedbefore the drive to ensure the users were familiar with andalso comfortable performing the task in the vehicle testbed.For each driver, there are two main consistencies in theprocess of data collection. First, at the beginning of the drive,the driver was verbally instructed with the list of secondary

IEEE Intelligent Vehicles Symposium 2014

Subject VideoTime(min)

# SamplesAnnotated

Environment Time

1 9:08 10115 Sunny 4pm2 10:05 4491 Sunny 5pm

TABLE IV: Driver activity recognition dataset collected.Training and testing is done using cross-subject cross-validation.

tasks to perform during the drive. Second, the drivers wereallowed to drive with control over what secondary task theywanted to perform and when they wanted to perform it.For instance, interaction with the radio was motivated bythe driver and not the experiment supervisor. Driving wasperformed in urban, high-traffic settings.

To ensure generalization of the learned models, all testingis performed by leave-one-subject-out cross validation, wherethe data from one subject is used for testing and the datafrom other subjects is used for training. We collected a headand hand dataset with the following statistics: 7429 samplesof two hands on the wheel region, 719 samples of handinteracting with the side rest, 679 samples of hand on the gearand 3039 samples of instrument cluster region interaction.Table IV shows the statistics of the entire dataset. As thevideos were collected in sunny settings in the afternoon, theycontain significant illumination variation that are both globaland local (shadows).

B. Evaluating Hand and Head Integration

Although head pose and landmark cues are generated atevery frame, they may be delayed in their correlation to theannotated hand activity. Nonetheless, integrating head cuescould improve detection in transition among regions as wellas to reduce false positives by increasing the likelihood ofa hand being present at one of the regions. Two featuresets are compared over a variable sized window of timeprevious to the current frame. If δ is the size of the timewindow, we can simply concatenate the time series over[(t − δ), . . . , (t)] in order to generate φφφ(t) (referred to astemporal concatenation) or we may summarize the timewindow using global statistics. In particular, we use themean, minimum, and maximum values of the window togenerate a set of head features. The second approach seemsto produce significantly better results as shown in Fig. 4.

For the three region classification problem, head pose andlandmark cues exhibit a distinctive pattern over the temporalwindow. A large window to include the initial glance beforereaching to the instrument cluster or the gear shift as wellas any head motions during the interaction significantlyimproves classification as shown in Fig. 5. Both the gearshift and instrument cluster (denoted as IC) benefit from theintegration.

IV. CONCLUSION

Automotive systems should be designed to operate quicklyand efficiently in order to assist the human driver. To that end,

−8 −6 −4 −2 00.75

0.8

0.85

0.9

0.95

Time Window (sec)

Norm

aliz

ed A

ccura

cy

Global Statistics

Temporal Concatenation

Fig. 4: Integration results for hand and head cues for the threeregion activity recognition (wheel, gear shift, instrumentcluster). The head features are computed over different sizedtemporal windows (see Section III-B).

.95 .03 .02

.23 .77

.08 .15 .77

WHEEL

GEAR

IC

WH

EEL

GEAR

IC

.85 .01 .14

1.0

.07 .05 .88

WHEEL

GEAR

IC

WH

EEL

GEAR

IC

(a) Hand Only (83%) (b) Hand+Head (91%)

Fig. 5: Activity recognition based on (a) hand only cues and(b) hand+head cue integration for the three region activityclassification. As head cues are common with instrumentcluster and gear shift interaction, a significant improvementin results is shown. IC stands for instrument cluster.

we investigated leveraging a multiprespective, multimodalapproach for semantic understanding of the driver’s state.A set of in-vehicle secondary tasks performed during on-road driving was utilized to demonstrate the benefit of suchan approach. The cues from two views, of the hand andof the head, were integrated in order to produce a morerobust activity classification. The analysis shows promise intemporal modeling of head and hand events. Future workwould extend the activity grammar to include additionalactivities of more intricate maneuvers and driver gestures.Combining the head pose with the hand configuration toproduce semantic activities can be pursued using temporalstate models, as in [20].

V. ACKNOWLEDGMENT

We acknowledge support of the UC Discovery Programand associated industry partners. We also thank our UCSDLISA colleagues who helped in a variety of important waysin our research studies. Finally, we thank the reviewers fortheir constructive comments.

IEEE Intelligent Vehicles Symposium 2014

REFERENCES

[1] R. Satzoda and M. M. Trivedi, “Automated drive analysis withforward looking video and vehicle sensors,” IEEE Trans. IntelligentTransportation Systems, to appear 2014.

[2] B. T. Morris and M. M. Trivedi, “Trajectory learning for activity under-standing: Unsupervised, multilevel, and long-term adaptive approach,”Pattern Analysis and Machine Intelligence, IEEE Transactions on,vol. 33, no. 11, pp. 2287–2301, 2011.

[3] T. A. Dingus, S. Klauer, V. Neale, A. Petersen, S. Lee, J. Sudweeks,M. Perez, J. Hankey, D. Ramsey, S. Gupta et al., “The 100-car natu-ralistic driving study, phase ii-results of the 100-car field experiment,”Tech. Rep., 2006.

[4] A. Doshi, B. Morris, and M. M. Trivedi, “On-road prediction ofdriver’s intent with multimodal sensory cues,” IEEE Pervasive Com-puting, vol. 10, no. 3, pp. 22–34, 2011.

[5] D. D. Waard, T. G. V. den Bold, and B. Lewis-Evans, “Driver handposition on the steering wheel while merging into motorway traffic,”Transportation Research Part F: Traffic Psychology and Behaviour,vol. 13, no. 2, pp. 129 – 140, 2010.

[6] S. Y. Cheng, S. Park, and M. M. Trivedi, “Multi-spectral and multi-perspective video arrays for driver body tracking and activity analysis,”Computer Vision and Image Understanding, vol. 106, no. 2, pp. 245–257, 2007.

[7] C. Tran and M. M. Trivedi, “3-d posture and gesture recognition forinteractivity in smart spaces,” Industrial Informatics, IEEE Transac-tions on, vol. 8, no. 1, pp. 178–187, 2012.

[8] S. Martin, A. Tawari, E. Murphy-Chutorian, S. Y. Cheng, andM. Trivedi, “On the design and evaluation of robust head pose forvisual user interfaces: Algorithms, databases, and comparisons,” inACM Conf. Automotive User Interfaces and Interactive VehicularApplications, 2012.

[9] E. Ohn-Bar, S. Martin, and M. M. Trivedi, “Driver hand activityanalysis in naturalistic driving studies: challenges, algorithms, andexperimental studies,” Journal of Electronic Imaging, vol. 22, no. 4,2013.

[10] S. Martin, A. Tawari, and M. M. Trivedi, “Towards privacy protectingsafety systems for naturalistic driving videos,” IEEE Trans. IntelligentTransportation Systems, 2014.

[11] E. Ohn-Bar, S. Martin, A. Tawari, and M. M. Trivedi, “Towards under-standing driver activities from head and hand coordinated movements,”in Pattern Recognition (ICPR), 2012 21st International Conference on.IEEE, 2012, pp. 605–608.

[12] K. Mathias and M. Turk, “Analysis of rotational robustness of handdetection with a viola-jones detector,” in Intl. Conf. on PatternRecognition, 2004.

[13] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan,“Object detection with discriminatively trained part-based models,”IEEE Trans. Pattern Analysis and Machine Intelligence,, vol. 32, no. 9,pp. 1627–1645, 2010.

[14] E. Ohn-Bar and M. M. Trivedi, “In-vehicle hand activity recognitionusing integration of regions,” in IEEE Conf. Intell. Veh. Symp., 2013.

[15] ——, “The power is in your hands: 3D analysis of hand gesturesin naturalistic video,” in IEEE Conf. Computer Vision and PatternRecognition Workshops, 2013.

[16] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vectormachines,” ACM Transactions on Intelligent Systems and Technology,vol. 2, pp. 27:1–27:27, 2011.

[17] A. Tawari, S. Martin, and M. M. Trivedi, “Continuous head movementestimator (cohmet) for driver assistance: Issues, algorithms and on-road evaluations,” IEEE Trans. Intelligent Transportation Systems,2014.

[18] X. Zhu and D. Ramanan, “Face detection, pose estimation, andlandmark localization in the wild,” in IEEE Conf. Computer Visionand Pattern Recognition, 2012.

[19] X. Xiong and F. D. la Torre, “Supervised descent method and itsapplications to face alignment,” in IEEE Conf. Computer Vision andPattern Recognition, 2013.

[20] Y. Song, L. P. Morency, and R. Davis, “Multi-view latent variablediscriminative models for action recognition,” in IEEE Conf. ComputerVision and Pattern Recognition, 2012.

IEEE Intelligent Vehicles Symposium 2014