human emotion synthesis david oziem, lisa gralewski, neill campbell, colin dalton, david gibson,...
TRANSCRIPT
Human Emotion Synthesis
David Oziem, Lisa Gralewski, Neill Campbell, Colin Dalton, David Gibson, Barry Thomas
University of Bristol, Motion Ripper, 3CR Research
Synthesising Facial Emotions – University of Bristol – 3CR Research
Project Group
• Motion Ripper Project
– Methods of motion capture.– Re-using captured motion signatures.– Synthesising new or extend motion sequences.– Tools to aid animation.
• Collaboration between University of Bristol CS, Matrix Media & Granada.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Introduction
• What is an emotion?
• Ekman outlined 6 different basic emotions.– joy, disgust, surprise, fear, anger and sadness.
• Emotional states relate to ones expression and movement.
• Synthesising video footage of an actress expressing different emotions.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Synthesising Facial Emotions – University of Bristol – 3CR Research
Video Textures
• Video textures or temporal textures are textures with motion. (Szummer’96)
• Schodl’00, reordered frames from the original to produce loops or continuous sequences.
– Doesn’t produce new footage.
• Campbell’01, Fitzgibbon’01, Reissell’01, used Autoregressive process (ARP) to synthesis frames.
Examples of Video Textures
Synthesising Facial Emotions – University of Bristol – 3CR Research
Autoregressive Process
• Statistical model
• Calculating the model involves working out the parameter vector (a1…an) and w.
• n is known as the order of the sequence.
y(t) = – a1y(t – 1) – a2y(t – 2) – … – any(t – n) + w.ε
Parameter vector (a1,…,an) Noise
Current value at time t
Synthesising Facial Emotions – University of Bristol – 3CR Research
Autoregressive Process
• Statistical model
• Increasing dimensionality of y drastically increases the complexity in calculating (a1…an).
y(t) = – a1y(t – 1) – a2y(t – 2) – … – any(t – n) + w.ε
Synthesising Facial Emotions – University of Bristol – 3CR Research
Autoregressive Process
PCA analysis of Sad footage in 2D
Secondary mode
Primary mode
• Principal Components Analysis is used to reduce number of dimensions in the original sequence.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Autoregressive Process
PCA analysis of Sad footage in 2D Generated sequence using an ARP
Secondary mode Secondary mode
Primary mode Primary mode
• Non-Gaussian Distribution is incorrectly modelled by an ARP.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Face Modelling
• Campbell’01, synthesised a talking head.
• Cootes and Talyor’00, combined appearance model.– Isolates shape and texture.
• Requires labelled frames.– Must label important features
on the face.
Labelled points
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined Appearance
Shape space
Hand Labelled video footage provides a point set which represents the shape space of the clip.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined Appearance
Shape space Texture space
Warping each frame into a standard pose, creates the texture space.
The standard pose is the mean position of the points.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined Appearance
Shape space Texture space
Combined spaceCombined space
Joining the shape and texture space and then re-analysing using PCA produces the combined space.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined Appearance
Shape space Texture space
Combined space
Reconstruction of the original sequence from the combined space.
Combined spaceCombined space
Synthesising Facial Emotions – University of Bristol – 3CR Research
Secondary mode
Primary mode
Combined Appearance
Combined Appearance sequence
Original sequence in 2D
Secondary mode
Primary mode
Change in distribution after applyingThe combined appearance technique
Synthesising Facial Emotions – University of Bristol – 3CR Research
Secondary mode
Primary mode
Combined Appearance
Generated SequenceOriginal sequence
Secondary mode
Primary mode
ARPmodelARP
model
• Visually the generated plot appears to have been generated using the same stochastic process as the original.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
• Combine the benefits of copying with ARP– New motion signatures.– Handles non-Gaussian distributions.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
Original inputOriginal input
Reduced inputReduced input
PCAPCA
• Important to reduce the complexity of the search process.• Need around 30 to 40 dimensions in this example.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
Original inputOriginal input
Reduced inputReduced input
Segmented inputSegmented inputPCAPCA Reduced segmentsReduced segmentsPCAPCA
• Temporal segments of between 15 to 30 frames.• Need to reduce each segment to be able to train ARP’s.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
Original inputOriginal input
Reduced inputReduced input
Segmented inputSegmented input Reduced segmentsReduced segmentsPCAPCA PCAPCA
ARPARP
Synthesised segmentsSynthesised segments
• Many of the learned models are unstable.• 10-20% are usable.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
Original inputOriginal input
Reduced inputReduced input
Segmented inputSegmented input Reduced segmentsReduced segmentsPCAPCA PCAPCA
ARPARP
Synthesised segmentsSynthesised segmentsSegment selectionSegment selection
Outputted SequenceOutputted Sequence
Synthesising Facial Emotions – University of Bristol – 3CR Research
Example
First mode
Time t
End of generated sequence.
Possible segments.
Compared section
Synthesising Facial Emotions – University of Bristol – 3CR Research
First mode
Time t
Example
Closest 3 segmentsare chosen.
Synthesising Facial Emotions – University of Bristol – 3CR Research
First mode
Time t
Example
The segment to be copied is randomly selected from the closest 3.
Synthesising Facial Emotions – University of Bristol – 3CR Research
First mode
Time t
Example
Segments are blended together using a small overlap and averaging the overlapping pixels.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Secondary mode
Primary mode
Secondary mode
Primary mode
Copying& ARPmodel
Copying& ARPmodel
PCA analysis of Sad footage in 2D
Generated sequence
Copying and ARP
• Potentially infinitely long.• Includes new novel motions.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Results (Angry)
Source Footage Copying with ARPCombined Appearance ARP
• Combined appearance produces higher resolution frames.
• Better motion from the copying and ARP approach
Synthesising Facial Emotions – University of Bristol – 3CR Research
Results (Sad)
Source Footage Copying with ARPCombined Appearance ARP
• Similar results as with the angry footage– Copied approach is less blurred due to the reduced variance.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Comparison Results
- Combined appearance - Segment copying
• Simple objective comparison.– Randomly selected temporal segments.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Comparison
• Perceptually is it better to have good motion or higher resolution.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined appearance Segment Copying with ARP
Synthesising Facial Emotions – University of Bristol – 3CR Research
Other potential uses
• Self Organising Map
• Uses combined appearance– as each ARP model provides a
minimal representation of the given emotion.
• Can navigate between emotions to create new interstates.
Angry Sad Happy
Synthesising Facial Emotions – University of Bristol – 3CR Research
Conclusions
• Both methods can produce synthesised clips of a given emotion.
• Combined appearance produces higher definition frames.
• Copying and ARPs generates more natural movements.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Questions