+ sridhar godavarthy july 01, 2010 defense of a masters thesis computer science and engineering...
TRANSCRIPT
![Page 1: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/1.jpg)
+
Sridhar GodavarthyJuly 01, 2010
Defense of a Masters ThesisComputer Science and EngineeringUniversity of South Florida
Microexpression Spotting in Video Using Optical Strain
![Page 2: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/2.jpg)
Microexpression Spotting in Video Using
Optical StrainSridhar Godavarthy
Examining Committee
Dmitry B. Goldgof, Ph.D. – Major Professor
Sudeep Sarkar, Ph.D. Rangachar Kasturi, Ph.D.
July 01, 2010
Defense of a Masters ThesisComputer Science and EngineeringUniversity of South Florida
![Page 3: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/3.jpg)
+ Minutes of the presentationMicroexpressions - “micro” expressions.
Goal: Detect “interesting” sequences containing μE.
Approach: optical flow + strain thresholding.
Result: True positive detection as high as 80%.Good performance on real time videos.
Conclusion: Novel system. Scope for improvement. Need more datasets.
![Page 4: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/4.jpg)
+
Introduction
![Page 5: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/5.jpg)
+ Expressions
Social emotion conveyance
Non verbal
Voluntary or involuntary
6 primary expressions
5
Anger Disgust Fear Happiness Sadness Surprise
![Page 6: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/6.jpg)
+ Microexpressions – What?
Subtle movements of the human body Observable Insufficient to convey emotion
Masking an expression
1/25th to 1/5th of a second
Almost impossible to suppress
07/01/2010
6
![Page 7: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/7.jpg)
+ Why?
Lie Detection
Pain detection for autistic and anaesthetized patients
Social signal processing( boredom/ concentration detection)
Psychological counseling.
07/01/2010
7
![Page 8: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/8.jpg)
+ State of the art
07/01/2010
8
Microexpression Research
Psychology Vision
Intent FACS classificationOptical FlowGabor FiltersANNsRule Based
![Page 9: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/9.jpg)
+ Objective
Design a preprocessing system that
Spots microexpressions.
Handles small translational and rotational
motion
Improves performance of existing systems
Greater weight to true positives.
07/01/2010
9
![Page 10: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/10.jpg)
+Some Fundamentals
Optic Flow : Vector representation of temporal changes
Strain: Relative deformation of material (skin)
Haar Classifier / Viola-Jones Face detector Cascade of weak classifiers OpenCV implementation Uses Haar rectangular features
![Page 11: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/11.jpg)
+ Brief overview of algorithm ( Main Idea)Skin deforms during an expression.
Deformation peaks at peak of expression
Duration of increased strain corresponds to microexpression
07/01/2010
11
~22 frames
~5 frames
Peak DetectionThresholding
Frames
Str
ain
Mag
nitu
de
Macro Expression
Micro Expression
![Page 12: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/12.jpg)
+
Algorithm
![Page 13: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/13.jpg)
+System Flow
Split Frames Face Detection & Alignment Optical Flow
Strain MapSplit into regionsStrain patterns and
Thresholding
![Page 14: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/14.jpg)
+ Face detection
Viola-Jones face detector-OpenCV implementation
![Page 15: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/15.jpg)
+ Face Alignment: Rotation
07/01/2010
15
![Page 16: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/16.jpg)
+System Flow
Split Frames Face Detection & Alignment Optical Flow
Strain MapSplit into regionsStrain patterns and
Thresholding
![Page 17: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/17.jpg)
+ Optical flow
07/01/2010
17
• Black and Anandan• Dense
MJ Black’s Matlab imlementation of OF
![Page 18: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/18.jpg)
+System Flow
Split Frames Face Detection & Alignment Optical Flow
Strain MapSplit into regionsStrain patterns and
Thresholding
![Page 19: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/19.jpg)
+ Facial strain
07/01/2010
19
![Page 20: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/20.jpg)
+System Flow
Split Frames Face Detection & Alignment Optical Flow
Strain MapSplit into regionsStrain patterns and
Thresholding
![Page 21: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/21.jpg)
+ Region Splitting
forehead(fh)
right cheek (rc)
left cheek (lc)
right mouth(rm)
left mouth(lm)
right eye(re)
below mouth(bm)
AUs not covered• Blink• Close eyes• Neck tightening• Nostril flare
Automated. Manual intervention if classifier fails
![Page 22: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/22.jpg)
+
Datasets & Results
![Page 23: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/23.jpg)
+Datasets
USF: IRB Canal9: EULA Found Videos: Fair use act
USF(100)
Canal9 (24) Found (4)
![Page 24: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/24.jpg)
+Threshold Determination
Sl. No.Threshold as percentage of
peak strain% True positives % False positives
175 31.8 0
250 50 0
335 77.2 22.7
430 54.5 36.4
525 13.6 0
07/01/2010
24
75 50 35 30 250
10
20
30
40
50
60
70
80
90
100
Threshold Selection
% Peak Strain
% T
rue
Po
sit
ive
![Page 25: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/25.jpg)
+Thresholded Strain Maps – Sample 1 / 3
07/01/2010
25
![Page 26: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/26.jpg)
+Thresholded Strain Maps – Sample 2 / 3
07/01/2010
26
![Page 27: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/27.jpg)
+Thresholded Strain Maps – Sample 3 / 3
07/01/2010
27
False Positive ( Indicative only)
![Page 28: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/28.jpg)
+Microexpression Spotting
07/01/2010
29
![Page 29: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/29.jpg)
+Negative Test Case – Rejects Expressions
07/01/2010
30
![Page 30: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/30.jpg)
+
Concluding Remarks
![Page 31: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/31.jpg)
+Contributions and Conclusions
Automated thresholding
Automated alignment (Partial)
Region wise detection
Up to 80% true detection
Microexpressions with expressions are detected.
![Page 32: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/32.jpg)
+Constraints
Constant illumination
Neutral face
Some expressions may be falsely detected
Talking
![Page 33: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/33.jpg)
+ Future Work
Dataset CollectionReal time questioning videos
Fully automated face alignmentBy matching optical flow vectors
Automatic identification of neutral face
Automatic portioning of facesAnthropomorphic landmark identification
07/01/2010
34
![Page 34: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/34.jpg)
+Related Publications
Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S., "Towards macro- and micro-expression spotting in video using strain patterns," Workshop on Applications of Computer Vision, 2009 pp:1-6
07/01/2010
35
![Page 35: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/35.jpg)
+ Other Publications by author Candamo, J., Kasturi, R., Goldgof, D., Godavarthy, S., "Detecting Wires in
Cluttered Urban Scenes Using a Gaussian Model, " to appear in Proceedings of International Conference on Pattern Recognition(ICPR 2010), Turkey, 2010
Godavarthy, S., Roomi, M. Md., “Adaptive Contrast Based Unsharp Masking,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002
Godavarthy, S., Pandian, A., Roomi, M. Md., “Histogram Equalization by Measure of Enhancement,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002
Godavarthy, S., Shankar, A., Roomi, M. Md., “Adaptive Watermarking-a FFT Approach”, Proceedings of International Conference on Advances in Telecommunication and Information Technology "Asia - Pacific Telecom 2000" (14th, 15th December 2000), Vellore
07/01/2010
36
![Page 36: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/36.jpg)
+Dr. Ekman on A-Rod
http://www.nytimes.com/2009/02/15/weekinreview/15marsh.html
![Page 37: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/37.jpg)
+
THANK YOU
38
![Page 38: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/38.jpg)
+Index Presentation:
Minutes
System Flow
Thresholding
Sample Strain Maps
Results
Negative Test Case
Conclusion and Future Work
Additional Slides
Evolutionary psychology
Detailed Flow Chart
Optical Flow
Elasticity and Strain
FACS
OF Vs OS
Dataset Details
![Page 39: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/39.jpg)
+
Additional Slides
![Page 40: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/40.jpg)
+
Study of everything we discussed until now
The child of ONE man - Paul Ekman. Over thirty years of research One of the world’s leading experts on lying. About 2 dozen books and innumerable articles Developed FACS Scientific Advisor to “Lie to Me” Co creator of Microexpression Training Tool (METTx)
Evolutionary Psychology
![Page 41: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/41.jpg)
+Flow Chart
07/01/2010
42
Split into individual frames
Detect and Crop face
Translate/Rotate to align with neutral mage
Calculate Robust Optical Flow
Calculate Optical Strain
Displacement Vectors
Split into Regions
Compute and normalize strain/region
Strain per pixel
Eight regions or less depending on visibility
Thresholding for magnitude, duration and spatial locality
Microexpression
Input video sequence
Viola-Jones Face Detector
Haar Cascade, Skin Detection
Finite Difference Method
Black and Anandan Method
Determine threshold for Strain Magnitude
Threshold
![Page 42: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/42.jpg)
+ Add: OFMotion Estimation: Optical Flow Method
Reflects the changes in the image due to motion
Computation is based on the following assumptions: observed brightness of any object point is constant over time nearby points in the image plane move in a similar manner
Minimization problem: (brightness const.) (smoothness const.)
Robust estimation framework (Black and Anandan, 1996) Recast the least squared formulations with a different error-norm function instead of
quadratic Coarse-to-fine strategy
Construct a pyramid of spatially filtered and sub-sampled images Compute flow values at lowest resolution and project to next level in the
pyramid
43
)()(),( 222222yxyxtyx vvuufvfufyxE
10/29/2009
![Page 43: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/43.jpg)
+
• Def: Optical Flow is the apparent motion of brightness patterns in the image
• Ideally, same as the motion field
• Have to be careful: apparent motion can be caused by lighting changes without any actual motion
Optical Flow
• Key assumptions• Brightness constancy: projection of the same
point looks the same in every frame• Small motion: points do not move very far• Spatial coherence: points move like their
neighbors
![Page 44: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/44.jpg)
Elasticity
Different materials have different elasticity
Elasticity can be modeled
strain
stressElasticity Known
Calculate
![Page 45: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/45.jpg)
+
What is Facial Strain? Strain on soft tissue when expressions are made. Anatomical method Uses a pair of frames to measure deformation
Facial Strain
![Page 46: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/46.jpg)
+
Finite Difference Method
Compute spatial derivatives from discrete points. Forward Difference Method Central Difference Method Richardson extrapolation
Strain Measurement
![Page 47: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/47.jpg)
+Thresholding
Threshold Strain Maps to segment out μE
07/01/2010
48
![Page 48: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/48.jpg)
+The Facial Action Coding System (FACS)
Coding of human expressions
Observational and Anatomical
32 Action Units and 14 Action Descriptors
Encode any possible [facial] expression.
Also used for facial expression simulations
![Page 49: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/49.jpg)
+ Name Method Type Content
Advantages Disadvantages
FAST O T S Early method Only negative emotions
FACS O A C All muscles Allows for
discovery
-
MAX O T S Faster performance
Only pre defined configurations.
EMG Obt A C Muscular activity invisible to naked eye
Interference from nearby muscles
EMFACS O T S Faster performance.
Only certain emotion expressions.
O - Observational Obt - Obtrusive A – Anatomical T – Theoretical S – Selective C – Comprehensive
![Page 50: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/50.jpg)
+FACS examples
07/01/2010
51
![Page 51: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/51.jpg)
+Why Optical Strain?
07/01/2010
52
![Page 52: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/52.jpg)
+ Datasets
Dataset Name
No. of Sequence
s
Approximate Duration
per sequence(s)
Microexpressions
per sequenc
e
Total Resolution
USF – feigned 12 140 8 96 SD / HD
USF – questioning 4 65 1 4 HD
Canal9 dataset 6 300-400 4 24 HD
Found videos 3 30-40 1 4 Very Low
07/01/2010
53
![Page 53: + Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida](https://reader038.vdocument.in/reader038/viewer/2022110205/56649cb25503460f94977775/html5/thumbnails/53.jpg)
+Index Presentation:
Minutes
System Flow
Thresholding
Sample Strain Maps
Results
Negative Test Case
Conclusion and Future Work
Additional Slides
Evolutionary psychology
Detailed Flow Chart
Optical Flow
Elasticity and Strain
FACS
OF Vs OS
Dataset Details