Deformable Facial Model Construction for non-rigid motion

tracking

Workflow

3D Face Reconstruction Methods1. Geometry-Based Methods

Assumptions: Faces have symmetrical shape and texture

Input: two face images which are orthogonal to each other

• The frontal photo provides the x and y coordinates .

• Side photo provides the y and z coordinates

• Reconstructed face is texture mapped using the blended texture form the

orthogonal photos.

Advantage:

1. Does not require 3D face database

Disadvantage

1. Depends on quality of the acquisition

2.Fail to produce accurate results for asymmetrical face geometry and

appearance

2. Stereo methods• Pixel correspondences are established between the two images to create the

disparity map. • The disparity map and distance between the two cameras are used to

compute the depth map.

Disadvantage• Performance is often affected by the environment conditions.

3. Shape from Motion models• Manually annotate 44 facial points on a face as input• Mark manually same feature points on a generic 3D model.• Cylindrical projection to map all 3D generic mesh points to 2D • Triangulate 2D feature points• Texture map the face onto 3D generic model • Morph the generic model to match the original.

DISADVANTAGE:• Requires more source information and the operation is relatively complex.

Face Models:

Features of Generic 3D face mesh: Candide v3.1.6 :

• 113 vertices

• All coordinates are between -1.0 to 1.0

• 184 faces/triangles and for each triangle, 3 vertices.

• Each action is implemented as a list of vertex displacements,describing the change in face geometry.

Advantages:

• Well-defined features

• Efficient Triangulation

Why we require Face Model?

• To interpret images of faces, it is important to have a model of how the face can appear.

• Changes can be broken down into two parts: changes in shape and changes in texture (patterns of pixel values) across the face.

Cylindrical Model

Advantages:

• Includes both circular and

elliptical cylinder.

• Copes up with large out of plane

rotation

• Robustness to initialization error

• Copes with self occlusions and

pose variations generated by large

head rotations.

• Simple

• Less computational load of a

fitting process

Disadvantages:

• Non-rigid motions cannot be

calculated as the vertices of the

model do not displace.

• Cannot generate actual shape and

texture

Ellipsoidal Model

• Ellipsoidal considers horizontally and vertical curved surfaces.• Accurately captures the 3D motion parameters of the head.• Is robust to small variations in the initial fit, enabling

the automation of the model initialization.• It considers the entire 3D aspect of the head, the

tracking is very stable over a large number of frames. Thisrobustness extends even to sequences with very low framerates and noisy camera images.

Planar Model• Plane model does not represent curved surfaces and is not

robust to out-of-plane rotations.

Figures taken from [12]

Facial deformable models

• Holistic modelsuses holistic texture based facial representation

Ex: AAM, 3D deformable models

# Discriminative

# Generative

• Part based modelsuses local image patches around landmark points

Ex: ASM, CLMs and Tree-based pictorial structures.

Slide Taken from:http://www.robots.ox.ac.uk/~minhhoai/papers/learn2align_CVPR08.pdf

Appearance Models

• Eigenfaces (Turk and Pentland, 1991)

– Not robust to shape changes

– Not robust to changes in pose and expression

• Ezzat and Poggio approach (1996)

– Synthesize new views of face from set of example views

– Does not generalize to unseen faces

Slide taken from: http://www.ai.mit.edu/courses/6.899/lectures/lecture17aam.ppt

Active Shape Models

• Point Distribution ModelTraining: Apply PCA to labeled imagesNew image

– Project mean shape– Iteratively modify model points to fit local neighborhood

Advantages and Disadvantage• ASM is relatively fast• ASM too simplistic; not robust when new images

are introduced• May not converge to good solution• Key insight: ASM does not incorporate all gray-

level information in parametersSlide taken from: http://www.ai.mit.edu/courses/6.899/lectures/lecture17aam.ppt

Slide Taken from :pages.cpsc.ucalgary.ca/~marina/601/Week6_Face_tracking.ppt

Example of ASM failing

The figure demonstrates the Active Shape Model (ASM) failing. The main facial features have been found, but the local models searching for the edges of the face have failed to locate their correct positions, perhaps because they are too far away. The ASM is a local method and prone to local minima.

Example of ASM search failure. The search profiles are not long enough to locate the edges of the face.

Combined Appearance Models

• Combine shape and gray-level variation in single statistical appearance model

• Advantages and Disadvantage

– Inherits appearance model benefits

• Able to represent any face within bounds of the training set

• Robust interpretation

– Model parameters characterize facial features

Slide taken from: http://www.ai.mit.edu/courses/6.899/lectures/lecture17aam.ppt

Parts Based Models

• Each part explains the image data underneath it.

• Model is represented as a graph.

• Vertices represents parts

• Edges represent connection between parts

• If we calculate best location for each part- we get connections for free.

Deformable model considers each object as a deformed version of a template leading to compact representation

Refer [9]

References

[1]Leung, W., Tseng, B., Shae, Z., Hendriks, F., and Chen, T. 2010. Realistic video avatar. Multimedia and Expo. IEEE.[2] CANDIDE – a parameterized face. http://www.bk.isy.liu.se/candide/main.html[3] Narendra Patel, Mukesh Zaveri," 3D Facial Model Construction and Expression Synthesis using a Single Frontal Face

Image”, International Journal of Graphics, November 2010[4] R. Valenti, N. Sebe, and T. Gevers, "Facial expression recognition: A fully integrated approach," in Int. Workshop on

Visual and Multimedia Digital Libraries, 2007[5] Iain Matthews, Jing Xiao, Simon Baker. “2D vs 3D Deformable Face Models: Representational Power, Construction

and Real-Time Fitting” International Journal of Computer Vision, Springer 2007.[6] P. Viola and M. Jones. Robust real-time object detection. International Journal of Computer Vision, 57(2):137–154,

2004.[7]M.Turk and A.Pentland. Eigen faces for recognition. Journal Cognitive Neuroscience, 3(1):71-86,1991[8] I. Matthews and S.Baker. Active appearance models revisited. International Journal of Computer Vision, 60(2):135-

164, Nov. 2004[9] Hamimah Ujir, “3D facial expresion classification using a statistical model of surface normals and modular approach,

theisus university of bBirmingham, 2012[10] K. H. An and M. Chung "3D head tracking and pose-robust 2D texture map-based face recognition using a simple

ellipsoid model", Proc. Intell. Robots Syst., pp.307 -312 2008 [11] Jung, Sung-Uk, and Mark S. Nixon. "On using gait biometrics to enhance face pose estimation." Biometrics: Theory

Applications and Systems (BTAS), 2010 Fourth IEEE International Conference on. IEEE, 2010.[12] S. Basu , I. A. Essa and A. P. Pentland "Motion regularization for model-based head tracking", International

Conference on Pattern Recognition, 1996 [13] La Cascia, M.; Sclaroff, S.; Athitsos, V., "Fast, reliable head tracking under varying illumination: an approach based

on registration of texture-mapped 3D models," Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol.22, no.4, pp.322,336, Apr 2000