lecture: face recognition - artificial...
Post on 23-Aug-2020
8 Views
Preview:
TRANSCRIPT
Face identification
Stanford University
05-Nov-2019
1
Lecture: Face Recognition
Juan Carlos Niebles and Ranjay KrishnaStanford Vision and Learning Lab
Face identification
Stanford University
05-Nov-2019
2
CS 131 Roadmap
Pixels Images
ConvolutionsEdgesDescriptors
Segments
ResizingSegmentationClustering
RecognitionDetectionMachine learning
Videos
MotionTracking
Web
Neural networksConvolutional neural networks
Face identification
Stanford University
05-Nov-2019
3
Let’s recap
• A simple object recognition pipeline with kNN• PCA
Face identification
Stanford University
05-Nov-2019
4
Object recognition: a classification framework
• Apply a prediction function to a feature representation of the image to get the desired output:
f( ) = “apple”f( ) = “tomato”f( ) = “cow”
Slide credit: L. LazebnikDataset: ETH-80, by B. Leibe
Face identification
Stanford University
05-Nov-2019
5
A simple pipeline - Training
Training Images
Image Features
Face identification
Stanford University
05-Nov-2019
6
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Face identification
Stanford University
05-Nov-2019
7
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Learned Classifier
Face identification
Stanford University
05-Nov-2019
8
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
Face identification
Stanford University
05-Nov-2019
9
Prediction
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
Learned Classifier
Face identification
Stanford University
05-Nov-2019
10
Prediction
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
Learned Classifier
Face identification
Stanford University
05-Nov-2019
11
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
Face identification
Stanford University
05-Nov-2019
12
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Face identification
Stanford University
05-Nov-2019
13
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Face identification
Stanford University
05-Nov-2019
14
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Face identification
Stanford University
05-Nov-2019
15
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation? Occlusion
Local shape: shape context Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Face identification
Stanford University
05-Nov-2019
16
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation? Occlusion
Local shape: shape context Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Face identification
Stanford University
05-Nov-2019
17
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation? Occlusion
Local shape: shape context Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Texture: Filter banks Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Face identification
Stanford University
05-Nov-2019
18
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation? Occlusion
Local shape: shape context Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Texture: Filter banks Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Face identification
Stanford University
05-Nov-2019
19
Prediction
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
Learned Classifier
Face identification
Stanford University
05-Nov-2019
20
Classifiers: Nearest neighbor
Training examples
from class 1
Training examples
from class 2
Slide credit: L. Lazebnik
Face identification
Stanford University
05-Nov-2019
21
Prediction
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
Learned Classifier
Face identification
Stanford University
05-Nov-2019
22
Classifiers: Nearest neighbor
Test example
Training examples
from class 1
Training examples
from class 2
Slide credit: L. Lazebnik
Face identification
Stanford University
05-Nov-2019
23
Let’s recap
• A simple object recognition pipeline with kNN• PCA
Face identification
Stanford University
05-Nov-2019
24
PCA compression: 144D -> 6D
Face identification
Stanford University
05-Nov-2019
25
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
6 most important eigenvectors
Face identification
Stanford University
05-Nov-2019
26
PCA compression: 144D ) 3D
Face identification
Stanford University
05-Nov-2019
27
2 4 6 8 10 12
2
4
6
8
10
122 4 6 8 10 12
2
4
6
8
10
12
2 4 6 8 10 12
2
4
6
8
10
12
3 most important eigenvectors
Face identification
Stanford University
05-Nov-2019
28
What we will learn today
• Introduction to face recognition• The Eigenfaces Algorithm• Linear Discriminant Analysis (LDA)
Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.
P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.
Face identification
Stanford University
05-Nov-2019
29Courtesy of Johannes M. Zanker
“Faces” in the brain
Face identification
Stanford University
05-Nov-2019
30
“Faces” in the brain fusiform face area
Kanwisher, et al. 1997
Face identification
Stanford University
05-Nov-2019
31
Detection versus Recognition
Detection finds the faces in images Recognition recognizes WHO the person is
Face identification
Stanford University
05-Nov-2019
32
Face Recognition
• Digital photography
Face identification
Stanford University
05-Nov-2019
33
Face Recognition
• Digital photography• Surveillance
Face identification
Stanford University
05-Nov-2019
34
Face Recognition
• Digital photography• Surveillance• Album organization
Face identification
Stanford University
05-Nov-2019
35
Face Recognition
• Digital photography• Surveillance• Album organization• Person tracking/id.
Face identification
Stanford University
05-Nov-2019
36
Face Recognition
• Digital photography• Surveillance• Album organization• Person tracking/id.• Emotions and expressions
Face identification
Stanford University
05-Nov-2019
37
Face Recognition
• Digital photography• Surveillance• Album organization• Person tracking/id.• Emotions and expressions• Security/warfare• Tele-conferencing• Etc.
Face identification
Stanford University
05-Nov-2019
38
The Space of Faces
• An image is a point in a high dimensional space– If represented in grayscale intensity,
an N x M image is a point in RNM
– E.g. 100x100 image = 10,000 dim
Slide credit: Chuck Dyer, Steve Seitz, Nishino
Face identification
Stanford University
05-Nov-2019
39
100x100 images can contain many things other than faces!
Face identification
Stanford University
05-Nov-2019
40
The Space of Faces
• An image is a point in a high dimensional space– If represented in grayscale intensity,
an N x M image is a point in RNM
– E.g. 100x100 image = 10,000 dim
• However, relatively few high dimensional vectors correspond to valid face images
• We want to effectively model the subspace of face images
Slide credit: Chuck Dyer, Steve Seitz, Nishino
ɸ1
Face identification
Stanford University
05-Nov-2019
41
Where have we seen something like this before?
ɸ1
Face identification
Stanford University
05-Nov-2019
42
Image space Face space
• Maximize the scatter of the training images in face space
• Compute n-dim subspace such that the projection of the data points onto the subspace has the largest variance among all n-dim subspaces.
Face identification
Stanford University
05-Nov-2019
43
• So, compress them to a low-dimensional subspace thatcaptures key appearance characteristics of the visual DOFs.
Key Idea
• USE PCA for estimating the sub-space (dimensionality reduction)
•Compare two faces by projecting the images into the subspace and measuring the EUCLIDEAN distance between them.
Face identification
Stanford University
05-Nov-2019
44
What we will learn today
• Introduction to face recognition• The Eigenfaces Algorithm• Linear Discriminant Analysis (LDA)
Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.
P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.
Face identification
Stanford University
05-Nov-2019
45
Eigenfaces: key idea
• Assume that most face images lie on a low-dimensional subspace determined by the first k (k<<d) directions of maximum variance
• Use PCA to determine the vectors or “eigenfaces” that span that subspace• Represent all face images in the dataset as linear combinations of eigenfaces
M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991
Face identification
Stanford University
05-Nov-2019
46
Training images: x1,…,xN
Face identification
Stanford University
05-Nov-2019
47
Top eigenvectors: ɸ1,…,ɸk
Mean: μ
Face identification
Stanford University
05-Nov-2019
48
Visualization of eigenfacesPrincipal component (eigenvector) ɸk
μ + 3σkɸk
μ – 3σkɸk
Face identification
Stanford University
05-Nov-2019
49
Eigenface algorithm
• Training1. Align training images x1, x2, …, xN
2. Compute average face
3. Compute the difference image (the centered data matrix)
Note that each image is formulated into a long vector!
µ =1N
xi∑
Face identification
Stanford University
05-Nov-2019
50
Eigenface algorithm
4. Compute the covariance matrix
5. Compute the eigenvectors of the covariance matrix Σ6. Compute each training image xi ‘s projections as
7. Visualize the estimated training face xi
xi → xic ⋅φ1, xi
c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )
xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK
Face identification
Stanford University
05-Nov-2019
51
Eigenface algorithm
6. Compute each training image xi ‘s projections as
7. Visualize the reconstructed training face xi
xi → xic ⋅φ1, xi
c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )
xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK
a1φ1 a2φ2 aKφK...Reconstructed training face
𝑥"
Face identification
Stanford University
05-Nov-2019
52
Eigenvalues (variance along eigenvectors)
Face identification
Stanford University
05-Nov-2019
53
• Only selecting the top K eigenfaces à reduces the dimensionality.• Fewer eigenfaces result in more information loss, and hence less
discrimination between faces.
Reconstruction and Errors
K = 4
K = 200
K = 400
Face identification
Stanford University
05-Nov-2019
54
Eigenface algorithm
• Testing1. Take query image t2. Project into eigenface space and compute projection
3. Compare projection w with all N training projections• Simple comparison metric: Euclidean• Simple decision: K-Nearest Neighbor
(note: this “K” refers to the k-NN algorithm, is different from the previous K’s referring to the # of principal components)
t→ (t −µ) ⋅φ1, (t −µ) ⋅φ2,..., (t −µ) ⋅φK( ) ≡ w1,w2,...,wK( )
Face identification
Stanford University
05-Nov-2019
56
Shortcomings
• Requires carefully controlled data:– All faces centered in frame– Same size– Some sensitivity to angle
• Alternative:– “Learn” one set of PCA vectors for each angle– Use the one with lowest error
• Method is completely knowledge free– (sometimes this is good!)– Doesn’t know that faces are wrapped around 3D objects (heads)– Makes no effort to preserve class distinctions
Face identification
Stanford University
05-Nov-2019
57
Summary for Eigenface
Pros• Non-iterative, globally optimal solution
Limitations
• PCA projection is optimal for reconstruction from a low dimensional basis, but may NOT be optimal for discrimination… Is there a better dimensionality reduction?
Face identification
Stanford University
05-Nov-2019
58
Besides face recognitions, we can also doFacial expression recognition
Face identification
Stanford University
05-Nov-2019
59
Happiness subspace (method A)
Face identification
Stanford University
05-Nov-2019
60
Disgust subspace (method A)
Face identification
Stanford University
05-Nov-2019
61
Facial Expression Recognition Movies (method A)
Face identification
Stanford University
05-Nov-2019
62
What we will learn today
• Introduction to face recognition• The Eigenfaces Algorithm• Linear Discriminant Analysis (LDA)
Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.
P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.
Face identification
Stanford University
05-Nov-2019
63
Which direction will is the first principle component?
Face identification
Stanford University
05-Nov-2019
64
Fischer’s Linear Discriminant Analysis
• Goal: find the best separation between two classes
Slide inspired by N. Vasconcelos
Face identification
Stanford University
05-Nov-2019
65
Difference between PCA and LDA
• PCA preserves maximum variance
• LDA preserves discrimination– Find projection that maximizes scatter between classes and minimizes scatter within
classes
Face identification
Stanford University
05-Nov-2019
66
Illustration of the Projection
Poor Projection
x1
x2
x1
x2
• Using two classes as example:
Good
Face identification
Stanford University
05-Nov-2019
67
Basic intuition: PCA vs. LDA
Face identification
Stanford University
05-Nov-2019
68
LDA with 2 variables• We want to learn a projection W such that the projection converts all the
points from x to a new space (For this example, assume m == 1):
• Let the per class means be:
• And the per class covariance matrices be:
• We want a projection that maximizes:
𝑧 = 𝑤&𝑥
Face identification
Stanford University
05-Nov-2019
69
Fischer’s Linear Discriminant Analysis
Slide inspired by N. Vasconcelos
Between class scatter
Within class scatter
Face identification
Stanford University
05-Nov-2019
70
LDA with 2 variablesThe following objective function:
Can be written as
Face identification
Stanford University
05-Nov-2019
71
LDA with 2 variables
• We can write the between class scatter as:
• Also, the within class scatter becomes:
Slide inspired by N. Vasconcelos
Face identification
Stanford University
05-Nov-2019
72
LDA with 2 variables
• We can plug in these scatter values to our objective function:
• And our objective becomes:
Slide inspired by N. Vasconcelos
Face identification
Stanford University
05-Nov-2019
73
LDA with 2 variables
• The scatter variables
Slide inspired by N. Vasconcelos
Face identification
Stanford University
05-Nov-2019
74
2S
1S
BS
21 SSSW +=
x1
x2Within class scatter
Between class scatter
Visualization
Face identification
Stanford University
05-Nov-2019
75
Linear Discriminant Analysis (LDA)
• Maximizing the ratio
• Is equivalent to maximizing the numerator while keeping the denominator constant, i.e.
• And can be accomplished using Lagrange multipliers, where we define the Lagrangian as
• And maximize with respect to both w and λ
Slide inspired by N. Vasconcelos
Face identification
Stanford University
05-Nov-2019
76
Linear Discriminant Analysis (LDA)• Setting the gradient of
With respect to w to zeros we get
or
• This is a generalized eigenvalue problem
• The solution is easy when exists
Slide inspired by N. Vasconcelos
Face identification
Stanford University
05-Nov-2019
77
Linear Discriminant Analysis (LDA)
• In this case
• And using the definition of SB
• Assuming that (μ1-μ0)Tw=α is a scalar, this can be written as
• and since we don’t care about the magnitude of w
Slide inspired by N. Vasconcelos
Face identification
Stanford University
05-Nov-2019
78
LDA with N variables and C classes
Face identification
Stanford University
05-Nov-2019
79
Variables• N Sample images:
• C classes:
• Average of each class:
• Average of all data:
{ }Nxx ,,1 !
å==
N
kkxN 1
1µ
å=Î ikx
ki
i xN c
µ 1
Face identification
Stanford University
05-Nov-2019
80
Scatter Matrices
• Scatter of class i:
å=
=c
iiW SS
1• Within class scatter:
• Between class scatter:
( )( )Tikx
iki xxSik
µµc
--= åÎ
𝑆( =)"*+
,
)-."
(𝜇" − 𝜇-)(𝜇" − 𝜇-)&
Face identification
Stanford University
05-Nov-2019
81
Mathematical Formulation• Recall that we want to learn a projection W
such that the projection converts all the points from x to a new space z:
• After projection:– Between class scatter– Within class scatter
• So, the objective becomes:
WSWS BT
B =~
WSWS WT
W =~
WSW
WSW
SS
WW
TB
T
W
Bopt WW
max arg~~
max arg ==
𝑧 = 𝑤&𝑥
Face identification
Stanford University
05-Nov-2019
82
Mathematical Formulation
• Solve generalized eigenvector problem:
WSW
WSWW
WT
BT
opt Wmax arg=
miwSwS iWiiB ,,1 !== l
Face identification
Stanford University
05-Nov-2019
83
Mathematical Formulation
• Solution: Generalized Eigenvectors
• Rank of Wopt is limited– Rank(SB) <= |C|-1– Rank(SW) <= N-C
miwSwS iWiiB ,,1 !== l
Face identification
Stanford University
05-Nov-2019
84
PCA vs. LDA
• Eigenfaces exploit the max scatter of the training images in face space
• Fisherfaces attempt to maximise the between class scatter, while minimising the within class scatter.
Face identification
Stanford University
05-Nov-2019
85
Basic intuition: PCA vs. LDA
Face identification
Stanford University
05-Nov-2019
86
Results: Eigenface vs. Fisherface
• Variation in Facial Expression, Eyewear, and Lighting
• Input: 160 images of 16 people• Train: 159 images• Test: 1 image
With glasses
Without glasses
3 Lighting conditions
5 expressions
Face identification
Stanford University
05-Nov-2019
87
Eigenface vs. Fisherface
Face identification
Stanford University
05-Nov-2019
88
What we have learned today
• Introduction to face recognition• The Eigenfaces Algorithm• Linear Discriminant Analysis (LDA)
Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.
P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.
top related