Micro expression Detection using Strain Patterns

-Sridhar Godavarthy

Based on

V.Manohar, D.B. Goldgof, S.Sarkar, Y. Zhang, "Facial Strain Pattern as a Soft Forensic Evidence", IEEE Workshop on Applications of Computer Vision (WACV'07),pp 42-42

Microexpressions

• What are microexpressions?– Subtle movements of the human body– Quick enough to be completed within the blink

of an eye– Not large enough to be called a micro

expression

Microexpressions Contd…

• Examples:• Raising an eyebrow• Shrugging of shoulders• Pout of lips• Fast blinking of eye

• Non Examples:• Talking• Smiling• Laughing• Anger

How do we go about it?

• Majority of the work “referenced” from : V.Manohar, D.B. Goldgof, S.Sarkar, Y. Zhang,

"Facial Strain Pattern as a Soft Forensic Evidence", IEEE Workshop on Applications of Computer Vision (WACV'07),pp 42-42

However, to keep things clear, we discuss each step in detail.

• Feature Based– Need to identify features – Difficult!– Features may be ill defined( when camouflaged)– Usually requires manual intervention– Produces a sparse motion field– Produce Good correspondence in large motion

• Optical Flow based– Fully automated– Dense Motion field.– Requires constant illumination

First Step – Obtaining Motion Field

• 3D Strain– Ideal– No high speed equipment available to capture

range images

• 2D Strain– Well – not much of a choice– Authors could use existing data.

Second Step – Strain Computation Type

• Finite Element Method– Forward modeling when Dirichlet condition is satisfied– Good at handling irregular shapes– Computationally expensive– This method is an approximation to the solution

• Finite Difference Method– Strain, a tensor, can be expressed derivatives of the displacement

vector– This can be approximated by a Finite Difference Method.– Very efficient when carried out on a regular grid.– This method is an approximation to the differential equation

Strain Computation - methods

• Finite Strain tensor

• Cauchy tensor

Finite Difference Method

Video

• Video is a collection of individual images also known as frames

• In reality: spatial and temporal compression using properties of the scene.

• Any video can be decoded into a series of frames.

• 24/30 frames per second of video.

Video Coding

• The science of encoding a video in a manner such that– Minimum number of

bits are used – Motion compensated

prediction can be performed from a previous frame.

Optical Flow• Pattern of apparent motion of objects/surfaces/edges

caused by relative motion between the observer(camera) and the scene

Elasticity• Different materials have different elasticity• Elasticity can be modeled

strain

stressElasticity Known

Calculate

Optical Strain• Variation of displacement values obtained from optical flow

– Calculated by taking the derivative of each pixel

Sobel operator (central difference)

Facial Strain

• What is Facial Strain?– Strain on soft tissue when expressions are

made.– Anatomical method– Uses a pair of frames to measure deformation

Strain Measurement

• Finite Difference Method• Compute spatial derivatives from discrete

points. – Forward Difference Method– Central Difference Method– Richardson extrapolation

The ProcessStart

Divide videos into Training and Testing Sequences

Read Training Video

Read Neutral

Micro Exp Macro

Exp Frames

Decode Video

Calculate OF & OS for Micro &

Macro frames

Training

Sequence

The Process Contd…Read Test

Video

Read Neutral Frames

Decode Video

Calculate Optical Flow between Neutral frame

and each frame

OF within desired range?

Reject Frame

Calculate Optical Strain between Neutral frame and this frame

OS within desired range?

Reject Frame

MICRO EXPRESSION

Stop

NoNo

Yes Yes

For Every other Frame

Test

Sequence

Program Output…

Program Output Contd…

RESULTS – Training & Test

Frame: Neutral Frame: Micro

Optical Flow Normalized optical flow

RESULTS - Test

Frame: Neutral Frame: Micro

Optical Flow Normalized optical flow

RESULTS - Test

Frame: Neutral Frame: Macro

Optical Flow Normalized optical flow

FALSE POSITIVE

RESULTS - Test

Frame: Neutral Frame: XYZ

Optical Flow Normalized optical flow

NOT DETECTED

Decisions and Consequences

• Use OF to reject highly improbable frames– Improves performance– Rejects frames with both Micro and Macro

Exp.– Segment image into regions?

• Threshold for classification was set to be ⅔T.O.S < M.O.S < 5/3 T.O.S

T.O.S = Training Optical Strain.

M.O.S = Measured Optical Strain

Possible Options

• Use only Optical Flow

• Use only Optical Strain

• Vary the thresholds

• Use an alternative for Max(OF/OS)

Future work

• Do not compare all frames( Skip frames immediately after a positive)

• Identify only one positive in a sequential list of positives

• Segment images to get separate Micro and Macro expressions

What I could not do

• Display strain as an image.• Segment image into regions and have

Regions of Interest

• V.Manohar, D.B. Goldgof, S.Sarkar, Y. Zhang, "Facial Strain Pattern as a Soft Forensic Evidence", IEEE Workshop on Applications of Computer Vision (WACV'07),pp 42-42

• Vasant Manohar, Matthew Shreve, Dmitry Goldgof and Sudeep Sarkar, "Finite Element Modeling of Facial Deformation in Videos for Computing Strain Pattern", International Conference on Pattern Recognition, Dec. 2008

• Matthew A. Shreve, Shaun J. Canavan, Yong Zhang, John R. Sullins, and Rupali Patil, "Imaging And Characterization Of Facial Strain In Long Video Sequences",xxxx

• Malcolm Gladwell,” Blink: The Power of Thinking Without Thinking”, Back Bay Books (April 3, 2007)

References

Thank You!

Sridhar Godavarthy

Dept. Of Computer Science and Engineering

University of South Florida

sgodavar@cse.usf.edu

About the TemplateFire was considered the most fearful of weapons in Greek Mythology. This template –

“Fireball” is a depiction of the might and power of fire, punching through ignorance and enlightening the audience