University of Washington The Hebrew University

* Microsoft Research

Synthesizing Realistic Facial Expressions from Photographs

Frederic Pighin Jamie Hecker Dani Lischinski David Salesin Richard Szeliski *

Realistic facial animationOne of the most challenging

problems in computer graphics:

•Facial surface is subtle•Facial movements are complex•Faces familiar

Muscles Bones[Parke and Waters, 96]

Our approach3 steps in synthesizing realistic faces:

1. Modeling from photographs2. Creating new expressions3. Animating expression changes

“surprised”

“worried”“sad”

“neutral” “joy”

Previous work: Modeling• Hand digitizing• 3D scanning• Image-based techniques

[Kleiser, 95]

Hand digitizing

[Lee et al., 95]

3D scanning

[Parke, 74]

Image-based techniques

Previous work: Animation• Performance-driven animation• Physically-based animation• 2D morphing• 3D morphing

[Williams, 90]

Performance-driven animation

Physically-based animation

[Lee et al., 95]

[Bregler et al., 97][Beier and Neeley, 92]

2D morphing

[Chen et al., 95]

3D morphing

1. Modeling from photographs

Overview of modeling1. Take multiple photographs of a person2. Establish corresponding feature points3. Recover 3D points and camera parameters 4. Deform generic face model to fit

points5. Extract textures from photos

Animating expression changesVarying the proportion of different

expressions over time creates animation

Pose recoverySolve for:

•3D locations of feature points•Camera orientation•Camera position•Focal length

Robust estimation by linear least squares

Mesh deformationInvolves two steps:

•Compute displacement of feature points

•Apply scattered data interpolant

Generic model DisplacementDeformed model

Texture map extractionThe color at each point is a weighted

combination of the colors in the photos

Texture can be:•View-independent •View-dependent

View-independent View-dependent

2. Creating new expressions

Creating new expressionsIn addition to global blending we can

use: •Regional blending•Painterly interface

“neutral”

“happy”

“fake smile”

+X

X

Design of a drunken smile

3. Animating expression changes