authors: sujitha martin, cuong tran, ashish tawari, jade...
TRANSCRIPT
![Page 1: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/1.jpg)
Presenter: Shinko Y. Cheng Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan, and Mohan M. Trivedi Date: Sept. 16th - 19th 2012 1
Presentation for IEEE Intelligent Transportation Systems Conference
![Page 2: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/2.jpg)
Introduction and Motivation
Key Terms and Research Issues
Related Studies
Optical flow based Head Movement and Gesture Analyzer (OHMeGA)
Concept and Algorithms
Noise and Other Practical Matters
Experimental Results Concluding Remarks
2
![Page 3: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/3.jpg)
Head pose is the 3D orientation of a head.
3
z
x
y
(yaw)
(pitch)
(roll)
![Page 4: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/4.jpg)
Head pose is the 3D orientation of a head. Head dynamics is the motion that describes
the change in head position
4
Head pose: Pitch = 0° Yaw = -30° Roll = 0°
Head pose: Pitch = 0° Yaw = +30° Roll = 0°
Head Dynamics: Pitch = 0° Yaw = +60° Roll = 0°
![Page 5: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/5.jpg)
Head pose is the 3D orientation of a head. Head dynamics is the motion that describes
the change in head position Head gesture entails how the head moved
from the starting orientation to the ending orientation.
5
![Page 6: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/6.jpg)
Safety of the driver and those in the vicinity is highly dependent on the driver’s awareness of the constantly changing driving environment
Head gesture detection and analysis is a vital part of looking inside a vehicle when designing intelligent driver assistance systems (IDAS).
6
![Page 7: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/7.jpg)
7
![Page 8: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/8.jpg)
Continuous head pose estimation for head gesture analysis is computationally intensive.
Higher level cues Fixation time and location Rate of motion and rate of change in motion
System goals for head gesture analysis in IDAS Runs in real-time User-independent Simple to implement and set-up Robust and accurate
8
![Page 9: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/9.jpg)
Feature vectors like head motion histograms (from head pose) for lane change intent prediction [1].
Head nodding frequency using head pose to determine driver vigilance[2].
Foot gesture analysis using optical flow in prediction
of driver behavior [3].
9
[1] B. Morris, A. Doshi and M. M. Trivedi, “Lane Change Intent Prediction for Driver Assistance: Design and On-Road Evaluation,” IEEE Intelligent Vehicles Symposium, 2011.
[2] L. M. Bergasa, J. Nuevo, M.A. Sotelo, R. Barea, and M.E. Lopez, “Real –time system for monitoring driver vigilance,” Intelligent Transportation Systems, IEEE Transactions on, 7(1):63-77, march 2006.
[3] C. Tran, A. Doshi, and M.M. Trivedi, “Modelling and Prediction of Driver Behavior by Foot Gesture Analysis”, Computer Vision and Image Understanding, 2012.
![Page 10: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/10.jpg)
10
• Intuitiveness: head gestures can be segmented into head motion states and no-head motion (fixation) states.
• Higher level cues: rate of head motion, rate of change in head motion, and fixation time.
![Page 11: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/11.jpg)
Rule based state machine
Two types of states Dynamic (motion) Static (no-motion or fixation)
Two parts Horizontal motion Vertical motion
Each set of colored arrows
represent the flow of one of four unique head gestures.
11
![Page 12: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/12.jpg)
12
• From a frontal facing camera, head motion in yaw and pitch rotation angle can be represented as motion in the vertical and horizontal direction of the 2D image plane.
• Steps to compute global flow vector:
• Interest point detection
• Lucas-Kanade’s optical flow algorithm
• u = -S-1d
•
• Majority vote on optical flow vectors
![Page 13: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/13.jpg)
Non-ideal conditions are:
Finite frame rate
Noise in the camera sensors (i.e. camera vibrations)
Motions detected by optical flow in the image plane may not be only due to head movements (i.e. hand movements near the face)
No-direct correspondence between head rotation in yaw (pitch)
angle to horizontal (vertical) motion in the image plane
13
![Page 14: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/14.jpg)
Head gesture: FxS ML FxL MR FxS
Top curve: horizontal
motion detected in the image plane with state labels
Bottom curve: ground truth state labels
Using threshold and area under the curve to handle noise
Prequal Exam OHMEGA in the Context of Driving 9/5/2012
14
![Page 15: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/15.jpg)
OHMeGA is evaluated on two sets of data
In-laboratory:
▪ 5 subjects (~600 head gestures),
▪ Subjects followed instructions (i.e. “STOP” and “GO”) by pressing the brake or the accelerator pedal
▪ Subjects answered “distractions” in the form of mathematical equations on the right side monitor.
On-road: manually selected head gestures for preliminary evaluations
15
![Page 16: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/16.jpg)
16
Ground truth labels
Global flow vector with labels
Motion in the x-direction of the image plane
Frame
•FxS •ML • FxL •MR
![Page 17: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/17.jpg)
FxS FxR FxL FxD
FxS 0.943 0 0 0
FxR 0 0.76 0 0
FxL 0 0 0.833 0
FxD 0 0 0 0.806
Samples 926 25 60 31
18
MR ML MD MU
MR 0.8 0 0 0
ML 0 0.917 0 0
MD 0 0 0.913 0
MU 0 0 0 0.907
Samples 145 120 69 86
Figure: Data collected using frontal facing camera from on-road experiment is processed first using optical flow to obtain head motions, then annotated using OHMeGA analyzer and finally separated into three types of gestures.
![Page 18: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/18.jpg)
OHMeGA is user-independent, simple to implement and set up, and runs in real-time
This implementation of OHMeGA relies only on head dynamics.
OHMeGA can derive higher level cues
such as fixation time and relative rate of motion
19
![Page 19: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/19.jpg)
Represent 3D head motion in the yaw and pitch rotation angle with both horizontal and vertical motions in the 2D image plane
Optimize global flow vector calculation for out of plane rotations (currently optimal for in plane movements).
20
![Page 20: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/20.jpg)
Colleagues in Laboratory for Intelligent and Safe Automobiles, UC San Diego
21
![Page 21: Authors: Sujitha Martin, Cuong Tran, Ashish Tawari, Jade Kwan,cvrr.ucsd.edu/scmartin/presentation/ITSC2012_presentation.pdf · Presenter: Shinko Y. Cheng Authors: Sujitha Martin,](https://reader034.vdocument.in/reader034/viewer/2022052022/6036dd38904fad4fb0287088/html5/thumbnails/21.jpg)
Any Questions?
22