Active Appearance Models for Face Detection

Rocío Cabrera, Guillaume Lemaître, Mojdeh Rastgoo

2

Presentation Outline

1/14/20113D Digitization - Active Appearance Models for Face Detection

Introduction Database Used

The IMM Face Database Models

Statistical Shape Models Statistical Models of

Appearance Active Appearance Models

Implementation Training Stage Testing Stage

Conclusions

3

Introduction

1/14/20113D Digitization - Active Appearance Models for Face Detection

Non-trivial Applications in Machine Vision “Understand” the presented images

Recover image structure Know what this structure means

Real applications include complex/variable structures Faces Detection

Model-based Methods Prior knowledge of the problem

Expected Shapes of Structures Their Spatial Relationship Greylevel Appearance

Restrict Automated Search to Plausible Interpretations

4

Introduction

1/14/20113D Digitization - Active Appearance Models for Face Detection

Generative Models Are able to generate realistic images of target objects

Deformable Models Are able to deal with object variability Two main desired characteristics

General – capable of generating plausible examples of the class they represent

Specific – capable of generating only legal/valid example

Model-based Methods Top-down Strategy

“Measure” if the target is actually present

Find Best Match in Image

Prior Model of Expected Class

5

The IMM Face Database

1/14/20113D Digitization - Active Appearance Models for Face Detection

An Annotated Dataset of 240 Face Images 40 different human subjects (7 females vs. 33 males)

All without glasses or accessories Manual Annotation of 58 landmarks

Six Different Positions Full frontal face, neutral/happy expression, diffuse light Face rotated (30° right/left), neutral expression, diffuse light Full frontal face, neutral expression, spot light added at the

person's left side. Full frontal face, arbitrary expression, diffuse light.

• Eyebrows• Eyes• Nose

• Mouth • Jaw

6

Statistical Shape Models

1/14/20113D Digitization - Active Appearance Models for Face Detection

Shape alignment

Modelling Shape Variation

Procrustes Analysis : Aligning the images onto the same reference axes Translation, Rotation and Scaling Transformations

Procrustes Analysis minimizes the distance between a reference shape and each shape in the dataset

Computation of the mean shape

Computation of the scatter (covariance) matrix

Sorting the eigenvectors and keeping the first k eigenvectors , based on the largest eigenvalues

Eigen decomposition of the shapes where ,

Value of k is based on

bxx

N

iixN

x1

1

N

i

Tii xxxx

Ns

1

))((1

1

i

},...,{ 21 k

k

ii

N

ivi f

11

98.0..10 gEfv

7

Statistical Shape Models

1/14/20113D Digitization - Active Appearance Models for Face Detection

Mean Shape and Largest Deformation

8

Statistical Shape Models

1/14/20113D Digitization - Active Appearance Models for Face Detection

9

Statistical Models of Appearance

1/14/20113D Digitization - Active Appearance Models for Face Detection

Texture mapping is required to generate the photo realistic synthetic images

Combination of a shape variation model with texture variation model

• Configuration of landmarks• Texture is the pattern of intensities or color across the image patch

shape model

Texture model

10

Statistical Models of Appearance

1/14/20113D Digitization - Active Appearance Models for Face Detection

Training set of label Images

Computation of statistical shape models -PCA

Computation of Free-Patch Images – Image wrapping

Applying PCA on Free-Patch Images

Statistical texture Models

Statistical Shape models – Mean shapes

Appearance models PCA

11

Statistical Models of Appearance – Image wrapping

1/14/20113D Digitization - Active Appearance Models for Face Detection

Piece Wise Affine Performing the Delaunay triangulation on each shape

model

Affine Transformation which maps the corner of the triangles to their new positions in new Image

12

Statistical Models of Appearance – Texture modeling

1/14/20113D Digitization - Active Appearance Models for Face Detection

Training set of shape-free normalized image patches

Performing PCA Model of texture:

},.....,{ 21 nggg

ggbPgg Mean normalized gray level

Set of orthogonal modes of variations

Set of gray level parameters

13

Statistical texture Models

1/14/20113D Digitization - Active Appearance Models for Face Detection

Eigen-faces decomposition

Texture model

14

Statistical Models of Appearance – Combined Image

1/14/20113D Digitization - Active Appearance Models for Face Detection

Shape parameter vector and texture parameter vector might have correlation

Performing PCA Appearance Model:

Controlling both shape and texture

Diagonal matrix of weight for each shape parameter

Eigenvectors

sb gb

)(

)(

ggP

xxPW

b

bWb T

g

Tss

g

ss

cPb c

Shape and texture will be a function of c

15

Statistical Appearance Models

1/14/20113D Digitization - Active Appearance Models for Face Detection

Texture model

Combination of texture model and shape model

Difference Image

16

Implementation

1/14/20113D Digitization - Active Appearance Models for Face Detection

Consists of two main stages Training Stage

Multi-scale implementation to obtain an AAM model N scales implementation

Testing Stage Searches for the object (face) in a test image

17

Implementation – Training Stage

1/14/20113D Digitization - Active Appearance Models for Face Detection

Load Training Data for SCALE = 1:N

Make Shape Model Align shapes with Procrustes Analysis Obtain main directions of variations with PCA Keep the 98% most significant eigenvectors

Grey-level appearance Model Transform face image into mean texture image Normalize the greyscale, to compensate for illumination Perform PCA Keep the 99% most significant eigenvectors

Combined Shape-Appearance Model Addition of the shape and appearance models Perform PCA Keep only 99% of all eigenvectors

Search Model Find the object location in a test set Training done by translation and intensity difference computation (keep position

with smallest difference) Transform the image to a coarser scale

end

Make Shape Model

Grey-level Appearance Model

Combined Shape-Appearance Model

Search Model

Transform Image to a Coarser Scale

18

Implementation – Testing Stage

1/14/20113D Digitization - Active Appearance Models for Face Detection

Manual Initialization For SCALE = 1:N (start in coarser scale)

Get Model for Current Scale Image Scaling Search Iterations

Sample Image Intensities Compute difference between model and real image intensities If Errorold < Errorcurrent

Go to previous location

Else Update Errorold

End End Go to next finer scale

End Show Detection Results

Search Iterations

Manual Initialization

Show Detection Results

19

Results

1/14/20113D Digitization - Active Appearance Models for Face Detection

Lower Scale

Higher Scale

20

Results

1/14/20113D Digitization - Active Appearance Models for Face Detection

Highest Scale

Texture map found at this scale

21

Problems Faced during Implementation

1/14/20113D Digitization - Active Appearance Models for Face Detection

Memory issues during the training

Problem of the reconstruction of the appearance

Not real-time application

22

Conclusion

1/14/20113D Digitization - Active Appearance Models for Face Detection

Face detection and face tracking are non-trivial applications in machine vision

Model-based methods Prior knowledge of the problem

Expected Shapes of Structures Their Spatial Relationship Grey-level Appearance

Active Appearance Models Are built from a set of training examples

Should account for class variabilty

They heavily rely on Principal Component/ Eigenvalue Analysis Through a search algorithm we seek to interpret a new target image with

the optimal model parameters which best describe the target image

The extension to Face Detection was not yet achieved but it is expected to work for the deliverable due date

The use of AAM seem like a promising method to perform face detection and/or recognition

23

References

1/14/20113D Digitization - Active Appearance Models for Face Detection

[1] Ginneken B. et al. "Active Shape Model Segmentation with Optimal Features", IEEE Transactions on Medical Imaging 2002. 

[2] T.F. Cootes, G.J Edwards, and C,J. Taylor "Active Appearance Models", Proc. European Conference on Computer Vision 1998 

[3] T.F. Cootes, G.J Edwards, and C,J. Taylor "Active Appearance Models", IEEE Transactions on Pattern Analysis and Machine Intelligence 2001

[4] Vazeos Ioannis. Active Appearance Models (AAM). MASTER THESIS REPORT. Master of Science in Information Networking. Athens Information Technology. 2004 -2005

[5] T.F. Gootes and C.J. Taylor Statistical Models of Appearance for Computer Vision. Imaging Science and Biomedical Engineering, University of Manchester. Technical Report. 2004.

[6] F. Dornaika and J. Ahlberg . Efficient Active Appearance Model for Real-Time Head and Facial Feature Tracking Proceedings of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures (AMFG’03). 2003

[7] Akshay Asthana, Jason Saragih, Michael Wagner and Roland Goecke. Evaluating AAM Fitting Methods for Facial Expression Recognition. 2009 IEEE

[8] Mingcai Zhou, Yangsheng Wang, Xiaoyan Wang and Xuetao Feng. A Two-Stage Approach for AAM Fitting. Eighth International Conference on Intelligent Systems Design and Applications. 2008 IEEE

[9] Fangqi Tang and Benzai Deng Facial Expression Recognition using AAM and Local Facial Features. Third International Conference on Natural Computation (ICNC 2007). 2007.