Faces: Analysis and Synthesis

Vision for GraphicsCSE 590SS, Winter 2001

Richard Szeliski

2/14/2001 Vision for Graphics 2

What can we do with faces?

Modeling (reconstruction):• manual

[Pighin et al. 1998]

• Automated[Zhang et al. 2000]

2/14/2001 Vision for Graphics 3

What can we do with faces?

Analysis• principal components and deformation modes

[Turk & Pentland 1991][Rowland & Perrett 1995][Guenter et al. 1998][Blanz & Vetter 1999]

2/14/2001 Vision for Graphics 4

What can we do with faces?

Tracking and synthesis• tracking

[Toyama 1998]• animation

[Pighin et al. 1999][Buck et al. 2000]

Recognition[Turk & Pentland 1991; Lanitis et al. 1997]

2/14/2001 Vision for Graphics 5

Synthesizing Realistic Facial Expressions from Photographs

Frederic Pighin Jamie Hecker Dani Lischinski

David Salesin Richard Szeliski *

SIGGRAPH’98

Animated Face Modeling From Video Images

Zhengyou Zhang, Zicheng Liu, Michael Cohen

Vision Group & Graphics Group

Microsoft Research

Manipulating Facial Appearance through Shape and Color

Duncan A. Rowland and David I. Perrett

St Andrews University

IEEE CG&A, September 1995

2/14/2001 Vision for Graphics 8

Principal component analysis

Compute average faces (color and shape)

Compute deviations between male and female (vector and color differences)

2/14/2001 Vision for Graphics 9

Changing gender

Deform shape and/or color of an input face in the direction of “more female”

original shape

color both

2/14/2001 Vision for Graphics 10

Enhancing gender

more same original androgynous more opposite

2/14/2001 Vision for Graphics 11

Changing age

Face becomes “rounder” and “more textured” and “grayer”

original shape

color both

A Morphable Model For The Synthesis Of 3D Faces

Volker Blanz Thomas Vetter

SIGGRAPH’99

2/14/2001 Vision for Graphics 13

Morphable model of 3D faces

Start with a catalogue of 200 3D Cyberware scans

Build a model of average shape and texture, and principal variations

2/14/2001 Vision for Graphics 14

Morphable model of 3D faces

Divide face into 4 regions (eyes, nose, mouth, head)

For each new prototype, find amount of deviation from the reference shape and texture.

2/14/2001 Vision for Graphics 15

Morphable model of 3D faces

Adding some variations

2/14/2001 Vision for Graphics 16

Reconstruction from single image

2/14/2001 Vision for Graphics 17

Modifying a single image

2/14/2001 Vision for Graphics 18

Animating from a single image

2/14/2001 Vision for Graphics 19

Resulting animation

Tracking Face Orientation

Kentaro Toyama

Vision–Based Interaction Group

Microsoft Research

2/14/2001 Vision for Graphics 21

1University of Washington 2Microsoft Research

ICCV’99

Resynthesizing Facial Animation through 3D Model-Based Tracking

Frédéric Pighin1 Richard Szeliski2 David Salesin1,2

Performance-Driven Hand-Drawn Animation

Ian Buck Adam Finkelstein

Charles Jacobs Allison Klein

David H. Salesin Joshua Seims Richard Szeliski Kentaro Toyama

2/14/2001 Vision for Graphics 23

Go from video to “cartoon”

2/14/2001 Vision for Graphics 24

Hand-drawn expressions

2/14/2001 Vision for Graphics 25

Expression correspondence

2/14/2001 Vision for Graphics 26

Morphing and tracking

2/14/2001 Vision for Graphics 27

More examples

2/14/2001 Vision for Graphics 28

Final animations

2/14/2001 Vision for Graphics 29

BibliographyF. Pighin, J. Hecker, D. Lischinski, D. H. Salesin, and R. Szeliski.

Synthesizing realistic facial expressions from photographs. In SIGGRAPH'98 Proceedings, pages 75--84, Orlando, July 1998.

Z. Liu, Z. Zhang, C. Jacobs, and M. Cohen. Rapid modeling of animated faces from video. Technical Report MSR-TR-2000-11, Microsoft Research, February 2000.

B. Guenter et al. Making faces. Proceedings of SIGGRAPH 98, pages 55--66, July 1998.

V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. Proceedings of SIGGRAPH 99, pages 187--194, August 1999.

K. Toyama. Prolegomena for robust face tracking. Technical Report MSR-TR-98-65, Microsoft Research, November 1998.

F. Pighin, D. H. Salesin, and R. Szeliski. Resynthesizing facial animation through 3D model-based tracking. In Seventh International Conference on Computer Vision (ICCV'99), pages 143--150, Kerkyra, Greece, September 1999.

2/14/2001 Vision for Graphics 30

BibliographyI. Buck et al. Performance-driven hand-drawn animation. In Symposium

on Non Photorealistic Animation and Rendering, pages 101--108, Annecy, June 2000. ACM SIGGRAPH.

D. A. Rowland and D. I. Perrett. Manipulating facial appearance through shape and color. IEEE Computer Graphics and Applications, 15(5):70--76, September 1995.

M. Turk and A. Pentland. Face recognition using eigenfaces. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'91), pages 586--591, Maui, Hawaii, June 1991. IEEE Computer Society Press.

P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman. Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):711--720, July 1997.

A. Lanitis, C. J. Taylor, and T. F. Cootes. Automatic interpretation and coding of face images using flexible models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):742--756, July 1997.