introduction our goal
DESCRIPTION
Automated Analysis of Interactional Synchrony using Robust Facial Tracking and Expression Recognition. Xiang Yu 1 , Shaoting Zhang 1 , Yang Yu 1 , Norah Dunbar 2 , Matthew Jensen 2 , Judee K. Burgoon 3 , Dimitris N. Metaxas 1 1 CBIM, Rutgers Univ., NJ - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/1.jpg)
Automated Analysis of Interactional Synchrony using Robust Facial Tracking and Expression
Recognition
Xiang Yu1, Shaoting Zhang1, Yang Yu1, Norah Dunbar2, Matthew Jensen2, Judee K. Burgoon3, Dimitris N.
Metaxas1
1CBIM, Rutgers Univ., NJ2Oklahoma University, Ok 3Univ. of Arizona, AZ
![Page 2: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/2.jpg)
IntroductionOur goal
• Predict whether an interactant is truthful or deceptive by analyzing interactional synchrony
• Propose a computational framework to do so
![Page 3: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/3.jpg)
IntroductionWhat is interactional synchrony?
• Interpersonal communication is contingent upon some form of mutual adaptation such as: – Accommodation– Interpersonal coordination– Matching– Mirroring– Compensation– Divergence– Complementarity
![Page 4: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/4.jpg)
IntroductionWhy investigating interactional synchrony?
• Practitioners have suggested using interactional synchrony for detecting deception:– with terrorists (Turvey, 2008)– in FBI interviews (Navarro, 2003)– and in police investigations (Kassin et al., 2007)
• Assumption: interviews with deceivers are less synchronous than interviews with truth tellers.
• However, few systematic studies of coordination, synchrony or reciprocity have examined the effects of synchrony on deception.
![Page 5: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/5.jpg)
IntroductionRelevant techniques
• Face Tracking - Active Shape Model (Cootes et al. CVIU 1995) - Active Appearance Model (Cootes et al. ECCV 1998) - Constrained Local Model (Cristinacce and Cootes, BMVC 2006)
• Gesture and Expression recognition - Head nodding, shaking, HMM(Kapoor and Picard PUI 2001) - Gesture recognition, CRF (Morency et al. CVPR 2007) - Image based expression recognition (Pantic et al. PAMI 2000) - Video based (Black and Yacoob, IJCV1997; Cohen et al. CVIU 2003)
![Page 6: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/6.jpg)
Proposed MethodSystem framework
![Page 7: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/7.jpg)
Module 1: Robust Facial Tracking
• Robust facial tracking is the foundation of interactional synchrony analysis
• Challenges:– Partial occlusion– Multiple poses– Poor lighting conditions
![Page 8: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/8.jpg)
Active Shape Models• Active Shape Model
Shape vector: Shape matrix: PCA: Covariance matrix: Shape representation:
![Page 9: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/9.jpg)
Active Shape Models
• Using KLT tracker to estimate shape locally.• Optimize shape , not only similar to , but also
follows the shape distribution:
• Alternately optimize and , until they converge.
![Page 10: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/10.jpg)
Limitations
• Partial occlusion:– Bayesian inference [Y. Zhou CVPR’03]– Pictorial structures [P. Felzenszwalb IJCV’05]– Sparse outliers [F. Yang FG’11]
• Multiple poses:– Mixture of Gaussian distribution [T. Cootes IVC’99]– Kernel PCA [S. Romdhani BMVC’07]– Hierarchical multi-state [Y. Tong PR’07]
![Page 11: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/11.jpg)
Face Tracking: Handle Occlusions
• We explicitly model the large errors as a sparse vector :
sparsity number of
![Page 12: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/12.jpg)
Face Tracking: Handle Multi-pose
• Instead of a low dimensional subspace, we model the shape as a sparse linear combination of training shapes:
≈...
![Page 13: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/13.jpg)
Face Tracking: Handle Multi-pose
• Instead of a low dimensional subspace, we model the shape as a sparse linear combination of training shapes:
sparsity number of sparsity number of
![Page 14: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/14.jpg)
Synthetic data
• Shape optimization from side pose with outliers. Blue line: detection results. Red line: fitting result.
E-ASM Sparse shape registration(Handle gross error)
Our method(Handle gross error
and multi-pose)
![Page 15: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/15.jpg)
Quantitative Comparison
• The point error by pixel
![Page 16: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/16.jpg)
Occlusion by Hat
Extended ASM Sparse Shape Registration Our method
![Page 17: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/17.jpg)
Occlusion by Scarf
Extended ASM Sparse Shape Registration Our method
![Page 18: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/18.jpg)
Multiple Pose
Sparse shape RegistrationOur Method
![Page 19: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/19.jpg)
Multiple Pose with Occlusions
Sparse shape RegistrationOur Method
![Page 20: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/20.jpg)
Face Tracking Results
• Tracking result from CMC dataset
![Page 21: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/21.jpg)
Module 2 and 3: Expression Detection and Head Pose
Demo of the successful detection of events (both are smiling and with the same head pose)
![Page 22: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/22.jpg)
Module 2: Expression Recognition
• Use the relative intensity order of facial expressions to learn a ranking model (RankBoost) for recog. and intensity estimation
• 6 universal facial expressions (i.e., fear, anger, sadness, disgust, happiness, surprise)
• Trained ranking model gives a score for the prob. of smiling in real time.
![Page 23: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/23.jpg)
Expression Recognition(1)
• Facial feature representation.
![Page 24: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/24.jpg)
Expression Recognition(2)
• Ordinal pair-wise data organization.
![Page 25: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/25.jpg)
Expression Recognition(3)
• Ranking Model. - sparse Rankboost (weak classifier) lost function:
- Adaboost strong classifier
![Page 26: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/26.jpg)
Smile Detection
Illustration of synchrony for a smiling event (both the subject and the interviewer are smiling at about the same time instant)
![Page 27: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/27.jpg)
Smile Synchrony
Left: smiling scores curves for both the subject (in blue) and interviewer (in red). Here we see that there is synchrony.Right: Snapshots from the actual tracked footage illustrating smiling synchrony.
![Page 28: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/28.jpg)
Module 3: Head Pose Module: Nodding
• A 3D head nodding sequence. The green lines show head pitch angle. The black lines show head yaw angle.
• Head pose pitch curve along time axis. The pattern of the blue plot is characteristic of head nodding
![Page 29: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/29.jpg)
Head Pose Module: Shaking• A 3D head shaking
sequence. The green lines show head pitch angle. The black lines show head yaw angle.
• head pose yaw curve. The pattern of the red plot is characteristic of a head shake.
![Page 30: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/30.jpg)
Head Pitch Synchrony (nodding)
head pitch curves for the subject (in blue) and the interviewer (in red). The patterns are characteristic of head nodding. Here we see that there is synchrony (both plots show nodding pattern).
![Page 31: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/31.jpg)
Head Yaw Dissynchrony (shaking)
head yaw curves for the subject (in blue) and the interviewer (in red). The distinct pattern of the red plot is characteristic of a head shake. This head shaking event is NOT in synchrony.
![Page 32: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/32.jpg)
Module 4: Synchrony Feature ExtractionCross correlation
• Generate higher level synchrony feature from Lower level features
• Correlation based strategy
![Page 33: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/33.jpg)
• Synchrony definition. – If event A is detected for subject X at
time T and for subject Y at time T ± w we say we have synchrony
• Some attributes can be extracted from detected Synchrony:– 1. Smile_Hit_Subject_Rate_S(x): Rate of
smiling synchrony (led by subject and followed by interviewer) in segment (x)
– 2. Smile_Hit_Interviewer_Rate_S(x): Rate of smiling synchrony (led by interviewer and followed by subject)
Smile_Hit_Subject
Smile
Time
Yes
No
SubjectInterviewer
Smile_Hit_Interviewer
Smile
Time
Yes
No
SubjectInterviewer
Synchrony Feature Extractionhit-miss rate
![Page 34: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/34.jpg)
Synchrony Feature Extractionhit-miss rate
• Some attributes can be extracted from detected Synchrony:– 3. Smile_Miss_Subject_Rate_S(x):
The rate of no smiling synchrony (led by subject and not followed by interviewer) in segment (x)
– 4.Smile_Miss_Interviewer_Rate_S(x): The rate of no smiling synchrony (led by subject and not followed by interviewer) in segment (x)
Smile_Miss_Subject
Smile_Miss_InterviewerSmile
Time
Yes
No
SubjectInterviewer
Smile
Time
Yes
No
SubjectInterviewer
![Page 35: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/35.jpg)
Module 5: Feature Selection and Classification
• Feature Selection, Genetic Algorithm.• Classification Two-class problem: Truthful vs. Deceptive
Three-class problem: Truthful, Sanctioned and Unsanctioned cheating
![Page 36: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/36.jpg)
Experimental Databases
• Computer-Mediated Communication Dataset (CMC)
• Face to Face Communication Dataset (FtF)
![Page 37: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/37.jpg)
Evaluation of Synchrony feature
![Page 38: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/38.jpg)
Evaluation of Two-class Classification
• Confusion Matrices
• Performance
![Page 39: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/39.jpg)
Evaluation of Three-class Classification
• Confusion Matrices
• Performance
![Page 40: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/40.jpg)
Conclusions
• We investigated how the degree of synchrony effects the result of deception detection.
• Automatic methods provide an important way to evaluate synchrony other than manual coding
• Some observations:– Different lower level features contribute differently
to the deception detection.–Modalities have subtle influence in detecting
deception. (CMC vs. FtF)
![Page 41: Introduction Our goal](https://reader036.vdocument.in/reader036/viewer/2022062315/56816499550346895dd6728f/html5/thumbnails/41.jpg)
Thanks!