Face Face IdentificationIdentificationby by FittingFitting a a

3D3D Morphable Model Morphable Modelusing using LinearLinear Shape and Texture Error Shape and Texture Error

FunctionsFunctions

Sami Romdhani Volker Blanz Thomas Vetter

University of Freiburg

Supported by DARPA

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 2/26

The ProblemThe Problem

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 3/26

MenuMenu

Historical Methods

3D Morphable Model

LiST : a Novel Fitting Algorithm

Identification Experiments on more than 5000 Images

Identification Confidence = Fitting Accuracy

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 4/26

Historical Methods : Historical Methods : Active Appearance ModelActive Appearance Model

Use of a generative model:

1. View based (2D), Correspondence basedex: AAM of Cootes and Taylor

Drawbacks:- small pose variation statistically

modeled !

- large pose var. necessitates many models !

- illumination not addressed !

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 5/26

Historical Methods : Illumination ConeHistorical Methods : Illumination Cone

2. Shape from Shading= Recovering 3D shape from Illumination variations

ex: Illumination Cone of Georghiades, Belhumeur & Kriegman

Limited use : up to 24° azimuth variation !

Drawback:Impractical: requires many imagesRestrictive assumptions : constant

albedo, lambertian,no cast shadows

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 6/26

3D Shape

3D Morphable Model - Key Features 13D Morphable Model - Key Features 1

1. Representation = 3D Shape + Texture Map

Texture Map

s

t

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 7/26

3D Morphable Model - Key Features 23D Morphable Model - Key Features 2

2. Accurate & Dense Correspondence

PCA accounts for intrinsic ID parameters only

2 3 4 1s S α

4 3 2 1t T β

...

...

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 8/26

3D Morphable Model - Key Features 33D Morphable Model - Key Features 3

3. Extrinsic parameters modeled using Physical Relations:- Pose : 3x3 Rotation matrix

- Illumination : Phong shading accounts for cast shadows and specular highlights

No Lambertian Assumption.

, ,

1 0 0

0 1 0xk

kk y

tx

yf

t

αR S

( )k

ambient dir speculark k k k

k

r

g

b

A TA aβ

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 9/26

3D Morphable Model - Key Features 43D Morphable Model - Key Features 4

4. Photo-realistic images rendered using Computer Graphicsα

S

, , R

1 0 0,

0 1 0f

,xt yt

, ,x y z

, ,x y z

,x y

β

T

, ,ambient dir specularaA A

, ,r g b

, ,r g b

( , , , , )w x y r g b ( , )mI x y

, ,x y zn n n

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 10/26

Model Fitting : DefinitionModel Fitting : Definition

IterativeModelFitting

,α β,ρ

ModelRenderin

g

( , )mI x y

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 11/26

Model Fitting - History : Standard Optimization Model Fitting - History : Standard Optimization TechniquesTechniquesJones, Poggio 98 : Gradient DescentBlanz, Vetter 99 : Stochastic Gradient DescentPighin, Szeliski, Salesin 99 : Levenberg-Marquardt

-

2

2

2

I

I

I

Model EstimateInput

Difference I

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 12/26

Model Fitting - History : Image Difference Model Fitting - History : Image Difference DecompositionDecomposition

IDD introduced by Gleicher in 97 and used by Sclaroff et al. in 98, and Cootes et al. in 98

-

I

A

Input

Difference I

Model Estimate

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 13/26

LiST : Non-linearity LiST : Non-linearity

1. Non-linear warping

2. Non-linear parametersinteraction

α

S

, , R

1 0 0,

0 1 0f

,xt yt

, ,x y z

, ,x y z

,x y

β

T

, ,ambient dir specularaA A

, ,r g b

, ,r g b

( , , , , )w x y r g b ( , )mI x y

, ,x y zn n n

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 14/26

LiST : Shape & Texture Parameters recoveryLiST : Shape & Texture Parameters recovery

α

S

, , R

1 0 0,

0 1 0f

,xt yt

,,x y z

, ,x y z

,x y

β

T

, ,ambient dir specularaA A

, ,r g b

, ,r g b

( , , , , )w x y r g b ( , )mI x y

, ,x y zn n n

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 15/26

( , , , , )w x y r g b

α

S

, , R

1 0 0,

0 1 0f

,xt yt

, ,x y z

, ,x y z

,x y

β

T

, ,ambient dir specularaA A

, ,r g b

, ,r g b

, ,x y zn n n

LiSTLiST

( , )mI x y

( , )I x y

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 16/26

Optical Flow

α

S

, , R

1 0 0,

0 1 0f

,xt yt

, ,x y z

, ,x y z

,x y

β

T

, ,ambient dir specularaA A

, ,r g b

, ,r g b

, ,x y zn n n

( , )mI x y

( , )I x y

LiST : Optical FlowLiST : Optical Flow

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 17/26

α

S

, , R

1 0 0,

0 1 0f

,xt yt

,,x y z

, ,x y z

,x y

β

T

, ,ambient dir specularaA A

, ,r g b

, ,r g b

, ,x y zn n n

Optical Flow( , )mI x y

( , )I x y

Lev.-Mar.

LiST : Rotation, Translation & Size RecoveryLiST : Rotation, Translation & Size Recovery

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 18/26

α

S

, , R

1 0 0,

0 1 0f

,xt yt

,,x y z

, ,x y z

,x y

β

T

, ,ambient dir specularaA A

, ,r g b

, ,r g b

, ,x y zn n n

Optical Flow( , )mI x y

( , )I x y

Lev.-Mar.Lev.-Mar.

LiST : Illumination RecoveryLiST : Illumination Recovery

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 19/26

LiST : DiscussionLiST : Discussion

• Shape and Texture recoveries are interleavedThe recovery of one helps the recovery of the other

• Takes advantage of the linear parts of the model

• Recovers out-of-the-image-plane rotation & directed illumination

• 5 times faster than Stochastic Gradient Descent

Drawbacks:• Still requires manual initialization• Still not fast enough

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 20/26

Experiments : The CMU-PIE Face DatabaseExperiments : The CMU-PIE Face Database

• Publicly available

• Systematic pose & illumination variations

• 68 Individuals

• 4488 Images with combined Pose & Illumination var.

• 884 Images with Pose var.

-20

-15

-10

-5

0 0

5

10

15

20

-5

0

5 head

flashescamerashead

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 21/26

Experiments : FittingExperiments : Fitting

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 22/26

Experiments : Identification across PoseExperiments : Identification across Pose

Identification Results across Pose

0102030405060708090

34 31 14 11 29 9 27 7 5 37 25 2 22

Gallery Pose

Pe

rce

nta

ge

LiST, average=76%FaceIt, average=21%

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 23/26

Experiments : Identification across Illumination & Experiments : Identification across Illumination & PosePose

Identification on 4488 imagesacross Pose & Illuminationaveraged over Illumination

Front Side Profile

Front 97 91 60

Side 93 96 71

Profile 65 71 86

Gallery

Probe

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 24/26

Identification Confidence : TheoryIdentification Confidence : Theory

Can we be sure to have correctly identified someone ?

Identification Confidence depends mostly on the Fitting

We think:

Classification Support Vector MachineInput:Mahalanobis distance from the average

SSE over 5 regions of the face

Output: Good Fitting Y/N ?

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 25/26

-2-1.25-0.75-0.250.250.751.2520

5

10

15

20

25

30

35

Fitting Score = SVM Output

% o

f Exp

erim

ents

29 % 33 % 12 % 6 % 4 % 7 % 7 % 3 %

Iden

tific

atio

n P

erce

ntag

e

Identification vs. Fitting Score

97.4 %95.1 %

83.7 %

76.5 %

58.9 %

43.2 %

38.2 %

26.8 %

20

30

40

50

60

70

80

90

100

Identification Confidence : ResultIdentification Confidence : Result

The model is goodwe only need to improve the fitting accuracy

7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 26/26

ConclusionsConclusions

Novel Fitting Algorithm :• Use of Optical Flow to recover a Shape Error• Recovers most of the parameters linearly• Recovers a few non-linear parameters using

Lev.-Mar.

State of the art identification performances across

Pose & Illumination

Drawbacks:• Still not fast enough• Still requires manual initialisation