lecture: face recognition - artificial...

Face identification

Stanford University

05-Nov-2019

1

Lecture: Face Recognition

Juan Carlos Niebles and Ranjay KrishnaStanford Vision and Learning Lab

Face identification

Stanford University

05-Nov-2019

2

CS 131 Roadmap

Pixels Images

ConvolutionsEdgesDescriptors

Segments

ResizingSegmentationClustering

RecognitionDetectionMachine learning

Videos

MotionTracking

Web

Neural networksConvolutional neural networks

Face identification

Stanford University

05-Nov-2019

3

Let’s recap

• A simple object recognition pipeline with kNN• PCA

Face identification

Stanford University

05-Nov-2019

4

Object recognition: a classification framework

• Apply a prediction function to a feature representation of the image to get the desired output:

f( ) = “apple”f( ) = “tomato”f( ) = “cow”

Slide credit: L. LazebnikDataset: ETH-80, by B. Leibe

Face identification

Stanford University

05-Nov-2019

5

A simple pipeline - Training

Training Images

Image Features

Face identification

Stanford University

05-Nov-2019

6

A simple pipeline - TrainingTraining LabelsTraining

Images

TrainingImage Features

Face identification

Stanford University

05-Nov-2019

7


Images


Learned Classifier

Face identification

Stanford University

05-Nov-2019

8


Images


Image Features

Test Image

Learned Classifier

Face identification

Stanford University

05-Nov-2019

9

Prediction


Images


Image Features

Test Image

Learned Classifier

Learned Classifier

Face identification

Stanford University

05-Nov-2019

10

Prediction


Images


Image Features

Test Image

Learned Classifier

Learned Classifier

Face identification

Stanford University

05-Nov-2019

11

Image featuresInput image

Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion

Face identification

Stanford University

05-Nov-2019

12


Color: Quantize RGB values Invariance?? Translation? Scale? Rotation (in-planar)

? Occlusion

Face identification

Stanford University

05-Nov-2019

13



? Occlusion

Global shape: PCA space Invariance?? Translation? Scale? Rotation (in-planar)

? Occlusion

Face identification

Stanford University

05-Nov-2019

14



? Occlusion

Global shape: PCA space Invariance?? Translation? Scale? Rotation (in-planar)

? Occlusion

Face identification

Stanford University

05-Nov-2019

15



Global shape: PCA space Invariance?? Translation? Scale? Rotation? Occlusion

Local shape: shape context Invariance?? Translation? Scale? Rotation (in-planar)

? Occlusion

Face identification

Stanford University

05-Nov-2019

16





? Occlusion

Face identification

Stanford University

05-Nov-2019

17





? Occlusion

Texture: Filter banks Invariance?? Translation? Scale? Rotation (in-planar)

? Occlusion

Face identification

Stanford University

05-Nov-2019

18





? Occlusion

Texture: Filter banks Invariance?? Translation? Scale? Rotation (in-planar)

? Occlusion

Face identification

Stanford University

05-Nov-2019

19

Prediction


Images


Image Features

Test Image

Learned Classifier

Learned Classifier

Face identification

Stanford University

05-Nov-2019

20

Classifiers: Nearest neighbor

Training examples

from class 1

Training examples

from class 2

Slide credit: L. Lazebnik

Face identification

Stanford University

05-Nov-2019

21

Prediction


Images


Image Features

Test Image

Learned Classifier

Learned Classifier

Face identification

Stanford University

05-Nov-2019

22

Classifiers: Nearest neighbor

Test example

Training examples

from class 1

Training examples

from class 2

Slide credit: L. Lazebnik

Face identification

Stanford University

05-Nov-2019

23

Let’s recap

• A simple object recognition pipeline with kNN• PCA

Face identification

Stanford University

05-Nov-2019

24

PCA compression: 144D -> 6D

Face identification

Stanford University

05-Nov-2019

25

2 4 6 8 10 12

24681012

2 4 6 8 10 12

24681012

2 4 6 8 10 12

24681012

2 4 6 8 10 12

24681012

2 4 6 8 10 12

24681012

2 4 6 8 10 12

24681012

6 most important eigenvectors

Face identification

Stanford University

05-Nov-2019

26

PCA compression: 144D ) 3D

Face identification

Stanford University

05-Nov-2019

27

2 4 6 8 10 12

2

4

6

8

10

122 4 6 8 10 12

2

4

6

8

10

12

2 4 6 8 10 12

2

4

6

8

10

12

3 most important eigenvectors

Face identification

Stanford University

05-Nov-2019

28

What we will learn today

• Introduction to face recognition• The Eigenfaces Algorithm• Linear Discriminant Analysis (LDA)

Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.

P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.

Face identification

Stanford University

05-Nov-2019

29Courtesy of Johannes M. Zanker

“Faces” in the brain

Face identification

Stanford University

05-Nov-2019

30

“Faces” in the brain fusiform face area

Kanwisher, et al. 1997

Face identification

Stanford University

05-Nov-2019

31

Detection versus Recognition

Detection finds the faces in images Recognition recognizes WHO the person is

Face identification

Stanford University

05-Nov-2019

32

Face Recognition

• Digital photography

Face identification

Stanford University

05-Nov-2019

33

Face Recognition

• Digital photography• Surveillance

Face identification

Stanford University

05-Nov-2019

34

Face Recognition

• Digital photography• Surveillance• Album organization

Face identification

Stanford University

05-Nov-2019

35

Face Recognition

• Digital photography• Surveillance• Album organization• Person tracking/id.

Face identification

Stanford University

05-Nov-2019

36

Face Recognition

• Digital photography• Surveillance• Album organization• Person tracking/id.• Emotions and expressions

Face identification

Stanford University

05-Nov-2019

37

Face Recognition

• Digital photography• Surveillance• Album organization• Person tracking/id.• Emotions and expressions• Security/warfare• Tele-conferencing• Etc.

Face identification

Stanford University

05-Nov-2019

38

The Space of Faces

• An image is a point in a high dimensional space– If represented in grayscale intensity,

an N x M image is a point in RNM

– E.g. 100x100 image = 10,000 dim

Slide credit: Chuck Dyer, Steve Seitz, Nishino

Face identification

Stanford University

05-Nov-2019

39

100x100 images can contain many things other than faces!

Face identification

Stanford University

05-Nov-2019

40

The Space of Faces

• An image is a point in a high dimensional space– If represented in grayscale intensity,

an N x M image is a point in RNM

– E.g. 100x100 image = 10,000 dim

• However, relatively few high dimensional vectors correspond to valid face images

• We want to effectively model the subspace of face images

Slide credit: Chuck Dyer, Steve Seitz, Nishino

ɸ1

Face identification

Stanford University

05-Nov-2019

41

Where have we seen something like this before?

ɸ1

Face identification

Stanford University

05-Nov-2019

42

Image space Face space

• Maximize the scatter of the training images in face space

• Compute n-dim subspace such that the projection of the data points onto the subspace has the largest variance among all n-dim subspaces.

Face identification

Stanford University

05-Nov-2019

43

• So, compress them to a low-dimensional subspace thatcaptures key appearance characteristics of the visual DOFs.

Key Idea

• USE PCA for estimating the sub-space (dimensionality reduction)

•Compare two faces by projecting the images into the subspace and measuring the EUCLIDEAN distance between them.

Face identification

Stanford University

05-Nov-2019

44





Face identification

Stanford University

05-Nov-2019

45

Eigenfaces: key idea

• Assume that most face images lie on a low-dimensional subspace determined by the first k (k<<d) directions of maximum variance

• Use PCA to determine the vectors or “eigenfaces” that span that subspace• Represent all face images in the dataset as linear combinations of eigenfaces

M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991

http://www.cs.ucsb.edu/~mturk/Papers/mturk-CVPR91.pdf

Face identification

Stanford University

05-Nov-2019

46

Training images: x1,…,xN

Face identification

Stanford University

05-Nov-2019

47

Top eigenvectors: ɸ1,…,ɸk

Mean: μ

Face identification

Stanford University

05-Nov-2019

48

Visualization of eigenfacesPrincipal component (eigenvector) ɸk

μ + 3σkɸk

μ – 3σkɸk

Face identification

Stanford University

05-Nov-2019

49

Eigenface algorithm

• Training1. Align training images x1, x2, …, xN

2. Compute average face

3. Compute the difference image (the centered data matrix)

Note that each image is formulated into a long vector!

µ =1N

xi∑

Face identification

Stanford University

05-Nov-2019

50

Eigenface algorithm

4. Compute the covariance matrix

5. Compute the eigenvectors of the covariance matrix Σ6. Compute each training image xi ‘s projections as

7. Visualize the estimated training face xi

xi → xic ⋅φ1, xi

c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )

xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK

Face identification

Stanford University

05-Nov-2019

51

Eigenface algorithm

6. Compute each training image xi ‘s projections as

7. Visualize the reconstructed training face xi

xi → xic ⋅φ1, xi

c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )

xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK

a1φ1 a2φ2 aKφK...Reconstructed training face

𝑥"

Face identification

Stanford University

05-Nov-2019

52

Eigenvalues (variance along eigenvectors)

Face identification

Stanford University

05-Nov-2019

53

• Only selecting the top K eigenfaces à reduces the dimensionality.• Fewer eigenfaces result in more information loss, and hence less

discrimination between faces.

Reconstruction and Errors

K = 4

K = 200

K = 400

Face identification

Stanford University

05-Nov-2019

54

Eigenface algorithm

• Testing1. Take query image t2. Project into eigenface space and compute projection

3. Compare projection w with all N training projections• Simple comparison metric: Euclidean• Simple decision: K-Nearest Neighbor

(note: this “K” refers to the k-NN algorithm, is different from the previous K’s referring to the # of principal components)

t→ (t −µ) ⋅φ1, (t −µ) ⋅φ2,..., (t −µ) ⋅φK( ) ≡ w1,w2,...,wK( )

Face identification

Stanford University

05-Nov-2019

56

Shortcomings

• Requires carefully controlled data:– All faces centered in frame– Same size– Some sensitivity to angle

• Alternative:– “Learn” one set of PCA vectors for each angle– Use the one with lowest error

• Method is completely knowledge free– (sometimes this is good!)– Doesn’t know that faces are wrapped around 3D objects (heads)– Makes no effort to preserve class distinctions

Face identification

Stanford University

05-Nov-2019

57

Summary for Eigenface

Pros• Non-iterative, globally optimal solution

Limitations

• PCA projection is optimal for reconstruction from a low dimensional basis, but may NOT be optimal for discrimination… Is there a better dimensionality reduction?

Face identification

Stanford University

05-Nov-2019

58

Besides face recognitions, we can also doFacial expression recognition

Face identification

Stanford University

05-Nov-2019

59

Happiness subspace (method A)

Face identification

Stanford University

05-Nov-2019

60

Disgust subspace (method A)

Face identification

Stanford University

05-Nov-2019

61

Facial Expression Recognition Movies (method A)

Face identification

Stanford University

05-Nov-2019

62





Face identification

Stanford University

05-Nov-2019

63

Which direction will is the first principle component?

Face identification

Stanford University

05-Nov-2019

64

Fischer’s Linear Discriminant Analysis

• Goal: find the best separation between two classes

Slide inspired by N. Vasconcelos

Face identification

Stanford University

05-Nov-2019

65

Difference between PCA and LDA

• PCA preserves maximum variance

• LDA preserves discrimination– Find projection that maximizes scatter between classes and minimizes scatter within

classes

Face identification

Stanford University

05-Nov-2019

66

Illustration of the Projection

Poor Projection

x1

x2

x1

x2

• Using two classes as example:

Good

Face identification

Stanford University

05-Nov-2019

67

Basic intuition: PCA vs. LDA

Face identification

Stanford University

05-Nov-2019

68

LDA with 2 variables• We want to learn a projection W such that the projection converts all the

points from x to a new space (For this example, assume m == 1):

• Let the per class means be:

• And the per class covariance matrices be:

• We want a projection that maximizes:

𝑧 = 𝑤&𝑥

Face identification

Stanford University

05-Nov-2019

69

Fischer’s Linear Discriminant Analysis


Between class scatter

Within class scatter

Face identification

Stanford University

05-Nov-2019

70

LDA with 2 variablesThe following objective function:

Can be written as

Face identification

Stanford University

05-Nov-2019

71

LDA with 2 variables

• We can write the between class scatter as:

• Also, the within class scatter becomes:


Face identification

Stanford University

05-Nov-2019

72


• We can plug in these scatter values to our objective function:

• And our objective becomes:


Face identification

Stanford University

05-Nov-2019

73


• The scatter variables


Face identification

Stanford University

05-Nov-2019

74

2S

1S

BS

21 SSSW +=

x1

x2Within class scatter

Between class scatter

Visualization

Face identification

Stanford University

05-Nov-2019

75

Linear Discriminant Analysis (LDA)

• Maximizing the ratio

• Is equivalent to maximizing the numerator while keeping the denominator constant, i.e.

• And can be accomplished using Lagrange multipliers, where we define the Lagrangian as

• And maximize with respect to both w and λ


Face identification

Stanford University

05-Nov-2019

76

Linear Discriminant Analysis (LDA)• Setting the gradient of

With respect to w to zeros we get

or

• This is a generalized eigenvalue problem

• The solution is easy when exists


Face identification

Stanford University

05-Nov-2019

77

Linear Discriminant Analysis (LDA)

• In this case

• And using the definition of SB

• Assuming that (μ1-μ0)Tw=α is a scalar, this can be written as

• and since we don’t care about the magnitude of w


Face identification

Stanford University

05-Nov-2019

78

LDA with N variables and C classes

Face identification

Stanford University

05-Nov-2019

79

Variables• N Sample images:

• C classes:

• Average of each class:

• Average of all data:

{ }Nxx ,,1 !

å==

N

kkxN 1

1µ

å=Î ikx

ki

i xN c

µ 1

Face identification

Stanford University

05-Nov-2019

80

Scatter Matrices

• Scatter of class i:

å=

=c

iiW SS

1• Within class scatter:

• Between class scatter:

( )( )Tikx

iki xxSik

µµc

--= åÎ

𝑆( =)"*+

,

)-."

(𝜇" − 𝜇-)(𝜇" − 𝜇-)&

Face identification

Stanford University

05-Nov-2019

81

Mathematical Formulation• Recall that we want to learn a projection W

such that the projection converts all the points from x to a new space z:

• After projection:– Between class scatter– Within class scatter

• So, the objective becomes:

WSWS BT

B =~

WSWS WT

W =~

WSW

WSW

SS

WW

TB

T

W

Bopt WW

max arg~~

max arg ==

𝑧 = 𝑤&𝑥

Face identification

Stanford University

05-Nov-2019

82

Mathematical Formulation

• Solve generalized eigenvector problem:

WSW

WSWW

WT

BT

opt Wmax arg=

miwSwS iWiiB ,,1 !== l

Face identification

Stanford University

05-Nov-2019

83

Mathematical Formulation

• Solution: Generalized Eigenvectors

• Rank of Wopt is limited– Rank(SB) <= |C|-1– Rank(SW) <= N-C

miwSwS iWiiB ,,1 !== l

Face identification

Stanford University

05-Nov-2019

84

PCA vs. LDA

• Eigenfaces exploit the max scatter of the training images in face space

• Fisherfaces attempt to maximise the between class scatter, while minimising the within class scatter.

Face identification

Stanford University

05-Nov-2019

85

Basic intuition: PCA vs. LDA

Face identification

Stanford University

05-Nov-2019

86

Results: Eigenface vs. Fisherface

• Variation in Facial Expression, Eyewear, and Lighting

• Input: 160 images of 16 people• Train: 159 images• Test: 1 image

With glasses

Without glasses

3 Lighting conditions

5 expressions

Face identification

Stanford University

05-Nov-2019

87

Eigenface vs. Fisherface

Face identification

Stanford University

05-Nov-2019

88

What we have learned today




lecture: face recognition - artificial...

Documents