perceptual and sensory augmented computing computer vision ws 08/09 computer vision – lecture 9...
TRANSCRIPT
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Computer Vision – Lecture 9Subspace Representations for
Recognition
3.12.2008
Bastian LeibeRWTH Aachenhttp://www.umic.rwth-aachen.de/multimedia
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Course Outline
• Image Processing Basics
• Segmentation & Grouping
• Recognition Global Representations Subspace representations
• Object Categorization I Sliding Window based Object Detection
• Local Features & Matching
• Object Categorization II Part based Approaches
• 3D Reconstruction
• Motion and Tracking2
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Image Segmentation
• Goal: identify groups of pixels that go together
3B. LeibeSlide credit: Steve Seitz, Kristen Grauman
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Images as Graphs
• Fully-connected graph Node (vertex) for every pixel Link between every pair of pixels, (p,q) Affinity weight wpq for each link (edge)
– wpq measures similarity
– Similarity is inversely proportional to difference (in color and position…)
4B. Leibe
q
p
wpq
w
Slide credit: Steve Seitz
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Normalized Cut (NCut)
• A minimum cut penalizes large segments• This can be fixed by normalizing for size of
segments• The normalized cut cost is:
• The exact solution is NP-hard but an approximation can be computed by solving a generalized eigenvalue problem.
5B. Leibe
assoc(A,V) = sum of weights of all edges in V that touch A
),(
),(
),(
),(
VBassoc
BAcut
VAassoc
BAcut
J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000
Slide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: NCuts: Overall Procedure
1. Construct a weighted graph G=(V,E) from an image.
2. Connect each pair of pixels, and assign graph edge weights,
3. Solve for the smallest few eigenvectors. This yields a continuous solution.
4. Threshold eigenvectors to get a discrete cut This is where the approximation is made (we’re not
solving NP).
5. Recursively subdivide if NCut value is below a pre-specified value.
6B. Leibe
( )D W y Dy
( , ) Prob. that and belong to the same region.W i j i j
Slide credit: Jitendra Malik
NCuts Matlab code available athttp://www.cis.upenn.edu/~jshi/software/
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Markov Random Fields
7B. Leibe
Image
Scene
Image patches
Scene patches
Slide credit: William Freeman
( , )i ix y
( , )i jx x
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Energy Formulation
• Joint probability
• Maximizing the joint probability is the same as minimizing the log
• This is similar to free-energy problems in statistical mechanics (spin glass theory). We therefore draw the analogy and call E an energy function.
and are called potentials.
8B. Leibe
,
( , ) ( , ) ( , )i i i ji i j
P x y x y x x
,
,
log ( , ) log ( , ) log ( , )
( , ) ( , ) ( , )
i i i ji i j
i i i ji i j
P x y x y x x
E x y x y x x
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Energy Formulation
• Energy function
• Single-node potentials Encode local information about the given pixel/patch How likely is a pixel/patch to belong to a certain class
(e.g. foreground/background)?
• Pairwise potentials Encode neighborhood information How different is a pixel/patch’s label from that of its
neighbor? (e.g. based on intensity/color/texture difference, edges)
9B. Leibe
Pairwisepotentials
Single-nodepotentials
( , )i ix y
( , )i jx x,
( , ) ( , ) ( , )i i i ji i j
E x y x y x x
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Graph Cuts Energy Minimization
10B. Leibe
pqw
n-links
s
t a cut)(tDp
t-lin
k
)(sDp
t-link
Regional bias exampleSuppose are given
“expected” intensities of object and background
ts II and 22 2/||||exp)( spp IIsD
22 2/||||exp)( tpp IItD
[Boykov & Jolly, ICCV’01]Slide credit: Yuri Boykov
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Graph Cuts Energy Minimization
11B. Leibe
pqw
n-links
s
t a cut)(tDp
t-lin
k
)(sDp
t-link
22 2/||||exp)( spp IIsD
22 2/||||exp)( tpp IItD
EM-style optimization
“expected” intensities of
object and background
can be re-estimated
ts II and
[Boykov & Jolly, ICCV’01]Slide credit: Yuri Boykov
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: When Can s-t Graph Cuts Be Applied?
• s-t graph cuts can only globally minimize binary energies that are submodular.
• Non-submodular cases can still be addressed with some optimality guarantees.
Current research topic12
B. Leibe
Npq
qpp
pp LLELELE ),()()(
},{ tsLp t-links n-links
Boundary termRegional term
E(L) can be minimized by s-t
graph cuts
),(),(),(),( stEtsEttEssE Submodularity (“convexity”)
[Boros & Hummer, 2002, Kolmogorov & Zabih, 2004]
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Topics of This Lecture
• Subspace Methods for Recognition Motivation
• Principal Component Analysis (PCA) Derivation Object recognition with PCA Eigenimages/Eigenfaces Limitations
• Fisher’s Linear Discriminant Analysis (LDA) Derivation Fisherfaces for recognition
• Robust PCA Application: robot localization
13B. Leibe
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Appearance-Based Recognition
• Basic assumption Objects can be represented
by a set of images(“appearances”).
For recognition, it is sufficient to just comparethe 2D appearances.
No 3D model is needed.
14
3D object
ry
rx
Fundamental paradigm shift in the 90’s
B. Leibe
=
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Recap: Recognition Using Global Features
• E.g. histogram comparison
15B. Leibe
Test image
Known objects
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Representations for Recognition
• More generally, we want to obtain representations that are well-suited for
Recognizing a certain class of objects Identifying individuals from that class (identification)
• How can we arrive at such a representation?
• Approach 1: Come up with a brilliant idea and tweak it until it
works.
• Can we do this more systematically?
16B. Leibe
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Example: The Space of All Face Images
• When viewed as vectors of pixel values, face images are extremely high-dimensional
100x100 image = 10,000 dimensions
• However, relatively few 10,000-dimensional vectors correspond to valid face images
• We want to effectively model the subspace of face images
17B. LeibeSlide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
The Space of All Face Images
• We want to construct a low-dimensional linear subspace that best explains the variation in the set of face images
18B. LeibeSlide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Subspace Methods
• Images represented as points in a high-dim. vector space
• Valid images populate only a small fraction of the space
• Characterize subspace spanned by images
19B. Leibe
… … …
Image setImage set Basis imagesBasis images RepresentationRepresentationcoefficientscoefficients
..
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Subspace Methods
• Today’s topics: PCA, LDA20
B. Leibe
Subspace Subspace methodsmethods
ReconstructiReconstructiveve
DiscriminativDiscriminativee
PCAPCA, ICA, , ICA, NMFNMF
LDALDA, SVM, , SVM, CCACCA
= +a= +a11 +a +a22 +a+a3 3 ++ ……
representatirepresentationon
classificaticlassificationon
regressionregression
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Topics of This Lecture
• Subspace Methods for Recognition Motivation
• Principal Component Analysis (PCA) Derivation Object recognition with PCA Eigenimages/Eigenfaces Limitations
• Fisher’s Linear Discriminant Analysis (LDA) Derivation Fisherfaces for recognition
• Robust PCA Application: robot localization
21B. Leibe
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Principal Component Analysis
• Given: N data points x1, … ,xN in Rd
• We want to find a new set of features that are linear combinations of original ones:
u(xi) = uT(xi – µ)
(µ: mean of data points)
• What unit vector u in Rd captures the most variance of the data?
22B. LeibeSlide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
• Direction that maximizes the variance of the projected data:
• The direction that maximizes the variance is the eigenvector associated with the largest eigenvalue of
Principal Component Analysis
23B. Leibe
Projection of data point
Covariance matrix of data
N
N
Slide credit: Svetlana Lazebnik
1
N
1
N
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
24
Remember: Fitting a Gaussian
• Mean and covariance matrix of data define a Gaussian model
g
1g
2g
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
25
Interpretation of PCA
• Compute eigenvectors of covariance, S• Eigenvectors: main directions• Eigenvalue: variance along eigenvector
• Result: coordinate transform to best represent the variance of the data
1 1 e2 2 e
g
1g
2g
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Properties of PCA
• It can be shown that the mean square error between xi and its reconstruction using only m principle eigenvectors is given by the expression:
• Interpretation PCA minimizes reconstruction error PCA maximizes variance of projection Finds a more “natural” coordinate system for the sample
data.26
B. Leibe
N
mjj
m
jj
N
jj
111
Slide credit: Ales Leonardis
Cumulative influenceof eigenvectors
90% of variance
k eigenvectors
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Projection and Reconstruction
• An n-pixel image xRn can beprojected to a low-dimensionalfeature space yRm by
• From yRm, the reconstruc-tion of the point is WTy
• The error of the reconstruc-tion is
27B. LeibeSlide credit: Peter Belhumeur
x 2 Rn
y Wx
Tx W Wx
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Example: Object Representation
28B. LeibeSlide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Principal Component Analysis
29B. Leibe
= + a1 + a2 + a3
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Object Detection by Distance TO Eigenspace
• Scan a window over the imageand classify the window as objector non-object as follows:
Project window to subspaceand reconstruct as earlier.
Compute the distance bet-ween and the reconstruc-tion (reprojection error).
Local minima of distance overall image locations objectlocations
Repeat at different scales Possibly normalize window intensity
such that ||=1.
30B. LeibeSlide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Obj. Identification by Distance IN Eigenspace
• Objects are represented as coordinates in an n-dim. eigenspace.
• Example: 3D space with points representing individual objects or a
manifold representing parametric eigenspace (e.g., orientation, pose, illumination).
• Estimate parameters by finding the NN in the eigenspace
31B. LeibeSlide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Parametric Eigenspace
• Object identification / pose estimation Find nearest neighbor in eigenspace [Murase & Nayar,
IJCV’95]32
B. LeibeSlide adapted from Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Eigenfaces: Key Idea
• Assume that most face images lie on a low-dimensional subspace determined by the first k (k<d) directions of maximum variance
• Use PCA to determine the vectors u1,…uk that span that subspace:x ≈ μ + w1u1 + w2u2 + … + wkuk
• Represent each face using its “face space” coordinates (w1,…wk)
• Perform nearest-neighbor recognition in “face space”
33B. Leibe
M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991Slide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Eigenfaces Example
• Training imagesx1,…,xN
34B. LeibeSlide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Eigenfaces Example
35B. Leibe
Top eigenvectors:u1,…uk
Mean: μ
Slide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Eigenface Example 2 (Better Alignment)
36B. LeibeSlide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Eigenfaces Example
• Face x in “face space” coordinates:
37B. Leibe
=
Slide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Eigenfaces Example
• Face x in “face space” coordinates:
• Reconstruction:
38B. Leibe
=
= +
µ + w1u1 + w2u2 + w3u3 + w4u4 + …x =
Slide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Summary: Recognition with Eigenfaces
• Process labeled training images: Find mean µ and covariance matrix Σ Find k principal components (eigenvectors of Σ) u1,…
uk
Project each training image xi onto subspace spanned by principal components:(wi1,…,wik) = (u1
T(xi – µ), … , ukT(xi – µ))
• Given novel image x: Project onto subspace:
(w1,…,wk) = (u1T(x – µ), … , uk
T(x – µ)) Optional: check reconstruction error x – x to
determine whether image is really a face Classify as closest training face in k-dimensional
subspace39
B. LeibeSlide credit: Svetlana Lazebnik
^
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Important Footnote
• Don’t really implement PCA this way! Why?
1. How big is ? nn, where n is the number of pixels in an image! However, we only have m training examples, typically
m<<n.
will at most have rank m!
2. You only need the first k eigenvectors
40B. LeibeSlide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Singular Value Decomposition (SVD)
• Any mn matrix A may be factored such that
• U: mm, orthogonal matrix Columns of U are the eigenvectors of AAT
• V: nn, orthogonal matrix Columns are the eigenvectors of ATA
: mn, diagonal with non-negative entries (1, 2,…,
s) with s=min(m,n) are called the singular values. Singular values are the square roots of the eigenvalues of
both AAT and ATA. Columns of U are corresponding eigenvectors!
Result of SVD algorithm: 12…s41
[ ] [ ][ ][ ]
TA U V
m n m m m n n n
Slide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
SVD Properties
• Matlab: [u s v] = svd(A) where A = u*s*v’
• r = rank(A) Number of non-zero singular values
• U, V give us orthonormal bases for the subspaces of A
first r columns of U: column space of A last m-r columns of U: left nullspace of A first r columns of V: row space of A last n-r columns of V: nullspace of A
• For d r, the first d columns of U provide the best d-dimensional basis for columns of A in least-squares sense
42B. LeibeSlide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Performing PCA with SVD
• Singular values of A are the square roots of eigenvalues of both AAT and ATA.
Columns of U are the corresponding eigenvectors.
• And
• Covariance matrix
• So, ignoring the factor 1/n, subtract mean image from each input image, create data matrix, and perform (thin) SVD on the data matrix.
43B. Leibe
1 11
nTT T
i i n ni
a a a a a a AA
1
1
( )( )n
Ti in
i
x x
Slide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Thin SVD
• Any m by n matrix A may be factored such that
• If m>n, then one can view as:
• Where ’=diag(1,2,…,s) with s=min(m,n), and
lower matrix is (n-m by m) of zeros.
• Alternatively, you can write:
• In Matlab, thin SVD is: [U S V] = svds(A) 44
B. Leibe
'
0
TA U V
' ' TA U V This is what you
should use!
Slide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Limitations
• Global appearance method: not robust to misalignment, background variation
• Easy fix (with considerable manual overhead) Need to align the training examples
45B. LeibeSlide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Limitations
• PCA assumes that the data has a Gaussian distribution (mean µ, covariance matrix Σ)
The shape of this dataset is not well described by its principal components
46B. LeibeSlide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Limitations
• The direction of maximum variance is not always good for classification
47B. LeibeSlide credit: Svetlana Lazebnik
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Topics of This Lecture
• Subspace Methods for Recognition Motivation
• Principal Component Analysis (PCA) Derivation Object recognition with PCA Eigenimages/Eigenfaces Limitations
• Fisher’s Linear Discriminant Analysis (LDA) Derivation Fisherfaces for recognition
• Robust PCA Application: robot localization
48B. Leibe
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Restrictions of PCA
• PCA minimizes projection error
• PCA is „unsupervised“ no information on classes is used
• Discriminating information might be lost49
B. Leibe
PCA projection
Best discriminatingprojection
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Fischer’s Linear Discriminant Analysis (LDA)
• LDA is an enhancement to PCA Constructs a discriminant subspace that minimizes
the scatter between images of the same class and maximizes the scatter between different class images
50B. LeibeSlide adapted from Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Mean Images
• Let X1, X2,…, Xc be the classes in the database and let each class Xi, i = 1,2,…,c have k images xj, j=1,2,…,k.
• We compute the mean image i of each class Xi as:
• Now, the mean image of all the classes in the database can be calculated as:
51B. Leibe
k
jji x
k 1
1
c
iic 1
1
Slide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Scatter Matrices
• We calculate the within-class scatter matrix as:
• We calculate the between-class scatter matrix as:
52B. Leibe
Tik
c
i XxikW xxS
ik
)()(1
Tii
c
iiB NS ))((
1
Slide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Visualization
53B. Leibe
Good separation
2S
1S
BS
21 SSSW
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Linear Discriminant Analysis (LDA)
• Maximize distance between classes
• Minimize distance within a class
• Criterion:
Sb … between-class scatter matrixSw … within-class scatter matrix
• Vector w is a solution of a generalized eigenvalue problem:
• Classification function:
54B. Leibe
Class 1
Class 2
w
x
x
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
LDA Computation
• Maximization of
is given by solution of generalized eigenvalue problem
• Defining we get
which is a regular eigenvalue problem.
• For the c-class case we obtain (at most) c-1 projections.
55B. Leibe
( )T
BT
W
w S wJ w
w S w
B WS w S w
12Bv S w
1 12 21B W BS S S v v
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Face Recognition Difficulty: Lighting
• The same person with the same facial expression, and seen from the same viewpoint, can appear dramatically different when light sources illuminate the face from different directions.
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Application: Fisherfaces
• Idea: Using Fisher’s linear discriminant to find class-
specific linear projections that compensate for lighting/facial expression.
• Singularity problem The within-class scatter is always singular for face
recognition, since #training images << #pixels
This problem is overcome by applying PCA first
57B. LeibeSlide credit: Peter Belhumeur [Belhumeur et.al. 1997]
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Fisherfaces: Experiments
• Variation in lighting
58B. LeibeSlide credit: Peter Belhumeur [Belhumeur et.al. 1997]
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Fisherfaces: Experiments
Slide credit: Peter Belhumeur [Belhumeur et.al. 1997]
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Fisherfaces: Experimental Results
60B. LeibeSlide credit: Peter Belhumeur [Belhumeur et.al. 1997]
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Fisherfaces: Experiments
• Variation in facial expression, eye wear, lighting
61B. Leibe [Belhumeur et.al. 1997]Slide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Fisherfaces: Experimental Results
62B. Leibe [Belhumeur et.al. 1997]Slide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Fisherfaces: Interpretability
• Example Fisherface for recognition “Glasses/NoGlasses“
63B. Leibe [Belhumeur et.al. 1997]Slide credit: Peter Belhumeur
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Topics of This Lecture
• Subspace Methods for Recognition Motivation
• Principal Component Analysis (PCA) Derivation Object recognition with PCA Eigenimages/Eigenfaces Limitations
• Fisher’s Linear Discriminant Analysis (LDA) Derivation Fisherfaces for recognition
• Application: robot localization
• Robust PCA Robust recognition under occlusion
64B. Leibe
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Application: Mobile Robot Localization
65B. Leibe
Panoramic camera Omnidirectional image
Unwrapped image
Parabolicmirror
Camera
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Environment Map
• Environment represented by a large number of views• Localization = recognition
66B. LeibeSlide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Compression with PCA
67B. LeibeSlide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Localization
68B. LeibeSlide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Localization Results
69B. LeibeSlide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Problems: Occlusion, Noise, Illumination
• Standard projection for estimation of parameters gives an arbitrary error in case of noise and occlusion
70B. LeibeSlide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Topics of This Lecture
• Subspace Methods for Recognition Motivation
• Principal Component Analysis (PCA) Derivation Object recognition with PCA Eigenimages/Eigenfaces Limitations
• Fisher’s Linear Discriminant Analysis (LDA) Derivation Fisherfaces for recognition
• Application: robot localization
• Robust PCA Robust recognition under occlusion
71B. Leibe
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Robust Estimation of PCA Coefficients
• Main idea: Instead of using the standard approach Select a subset of pixels Find a robust solution of equations Evaluate multiple hypotheses
Hypothesize-and-test paradigm Competing hypotheses are subject to a selection
procedure based on the MDL principle
72B. Leibe
= = + + aa11 + + aa22 + + aa3 3
+ …+ …
= = + + aa11 + + aa22 + + aa3 3
+ …+ …
= = + + aa11 + + aa22 + + aa3 3
+ …+ …
……
linear system of equations
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Robust PCA Example
73B. Leibe
NNon-occluded on-occluded training imagestraining images
OOccluded imagesccluded images
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Robust PCA Example
74B. Leibe
RReconstructed econstructed non-occluded non-occluded
imagesimages
NNon-robustly on-robustly reconstructed reconstructed occluded imagesoccluded images
RRobustly obustly reconstructed reconstructed occluded imagesoccluded images
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Robust PCA Example
75B. Leibe
RReconstructed econstructed non-occluded non-occluded
imagesimages
NNon-robustly on-robustly reconstructed reconstructed occluded imagesoccluded images
RRobustly obustly reconstructed reconstructed occluded imagesoccluded images
GTGT occl.occl. rec.Grec.GTT
non-non-rob.rob.
rob.rob.
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Hypothesis Selection
• Three cases1. One object:
– Select best match (cii)
2. Multiple non-overlapping objects: – Select local maximum (cii) in sampling window
3. Multiple overlapping objects: – This is the only critical case.– Here, measurements from different objects may result in
spurious estimation results.– This can be resolved by a model selection criterion
76B. Leibe
A. Leonardis, H. Bischof, Robust Recognition Using Eigenimages, CVIU, Vol. 78(1), 2000.
Slide credit: Ales Leonardis
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
Robust Localization at 60% Occlusion
77B. Leibe
Standard approach Robust approach
Slide credit: Ales Leonardis
Mean localization error
Perc
ep
tual an
d S
en
sory
Au
gm
en
ted
Com
pu
tin
gC
om
pu
ter
Vis
ion
WS
08
/09
References and Further Reading
• Background information on PCA/LDA can be found in Chapter 22.3 of
D. Forsyth, J. Ponce,Computer Vision – A Modern Approach.Prentice Hall, 2003
• Important Papers (available on webpage) M. Turk, A. Pentland
Eigenfaces for RecognitionJ. Cognitive Neuroscience, Vol. 3(1), 1991.
P.N. Belhumeur, J.P. Hespanha, D.J. KriegmanEigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection, IEEE Trans. PAMI, Vol. 19(7), 1997.
A. Leonardis, H. Bischof,Robust Recognition using Eigenimages, CVIU, Vol. 78(1), 2000.