facial animation wilson chang paul salmon april 9, 1999 computer animation university of...

Facial AnimationWilson ChangPaul SalmonApril 9, 1999

Computer AnimationUniversity of Wisconsin-Madison

Papers Used Bregler C.,Covell M.,Slaney M., Video Rewrite: Driving Visual

Speech with Audio. In SIGGRAPH 97 Conference Proceedings. ACM SIGGRAPH, August 1997

Guenter B.,Grimm C.,Wood D., Malvar H., Pighin F, Making Faces. In SIGGRAPH 98 Conference Proceedings. ACM SIGGRAPH, July 1998

Pighin F, Hecker J., Lischinski D., Szeliski R., Salesin D., Synthesizing Realistic Facial Expressions from Photographs. In SIGGRAPH 1998.

Waters K., A Muscle Model for Animating Three-Dimensional Facial Expression. In SIGGRAPH 1987.

Motivation

Creation of Virtual CharactersTeleconferencing & Video CompressionSimulated MovementFacial Surgery Planning

Why facial animation is hard.Humans are very good

at reading expressions.Any slight deviation

from a “correct” expression will be immediately noticed.

Deep-rooted instinct.

Three general catagories

2-D Facial Model3-D Facial ModelMuscular Model

2-D Facial Animation

Video Rewrite - modify and sync an actors’ lip motion to a new soundtrack.

Keyframe approach.Uses vision techniques to track mouth

movement.

Video Rewrite registration

Hand annotation of 26 images with 54 eigenpoints each.

Morph pairs to 351 images.Learn eigenpoint model.Warp images to standard reference plane.Eigenpoint analysis.

Audio Analysis

Video Rewrite uses TIMIT speech database.Triphones - emphasize middle.“teapot” = /SIL-T-IY/, /T-IY-P/, /IY-P-AA/,

/P-AA-T/, /AA-T-SIL/

Video SynthesisTriphone Footage selection

error = Dp + (1- )Ds

Dp phoneme-context distance.

Ds distance between lip shapes. Overall Lip Width & Height Inner Lip Height Height of Visible Teeth

Finish Synthesis

Compress and Stretch video.Align and blend mouth to face.

Results

Good Sync and natural articulation.Missing Triphones result in unnatural

speech

E_noreg.mov S_green.movS_wish.mov jfk-forestgump.mov

Making Faces

Motion capture.3D mesh via Cyberware Laser scanner.Deformed by

Position of 128 Dots• Manual identification - 1st frame• Tracked by vision techniques

Texture Extraction Dot removal. Cylindrical map.

Synthesizing Realistic Facial Expressions from Photographs

3D facial models derived from photographs.Smooth transitioning between model

expressions.Adaptation from one model to another.

Model Fitting

Generic 3D mesh model.Pose Recovery - using

multiple subject views: Identify feature points. Deduce camera pose. Iteratively refine the generic

face model.

Model Fitting

Scattered Data Interpolation: Interpolate mesh between feature points. Uses radial basis functions.

Correspondence based shape refinement: Use less accurate correspondences. Polylines for eyebrows, eyelids, lips, etc. Not used in pose processing due to error.

Texture Extraction

View independent vs View dependent.Weight maps- bias selection of original

photograph: Self-occlusion. Smoothness. Positional certainty. View similarity.

View Dependent Texture ExtractionSelect best photographs.Draw model for each photograph.Blend rendered image.Pros

adds detail.Cons

sensitive to original photo. More memory, slower.

View Independent Texture Extraction

Blend photographs to form single texture. Map onto virtual cylinder.

View Independent Texture Extraction

Blurry

Special Case Textures

Fine Detail - hair.Occlusion - eyes, teeth.Intricate Projection - ears. Shadowing - eyes, teethSolutions

Use photo with highest visibility. Simulate shadowing

Expression MorphingSimplified by common mesh.Linearly interpolated vertices.Blend result of rendering with each texture.

Synthesize new expressions via: Global blend. Regional blend. Painterly interface.

ResultsSmooth transitioned expressions:

ResultsApplied transitions to different human

subject:

Our conclusions

Good results between models.Relatively inexpensive equipment.Notable manual processing.

Muscular Modeling

Easy generalized across models.22 muscle groupsFacial Action Coding System (Ekman,

Wallace) - Action Unit parameterization

Anatomy

Skin as Mesh

Nodal mobility Tensile Strength of skin Proximity to muscle attachment Depth of tissue & proximity to bone Elasticity & interaction with other muscles

Network of springs p = F/k

Mesh expression examples

Muscle types modeled

Linear/parallel musclesSphincter muscles

Linear/parallel muscles

Sphincter muscles

Animating

Not in paperBuild a libraryAbstract languageKeyframe