lec 15: graph embedding & laplacianface · pca is a special case of graph embedding ofully...
TRANSCRIPT
![Page 1: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/1.jpg)
Spring 2020: Venu: Haag 315, Time: M/W 4-5:15pm
ECE 5582 Computer VisionLec 15: Graph Embedding & Laplacianface
Zhu LiDept of CSEE, UMKC
Office: FH560E, Email: [email protected], Ph: x 2346.http://l.web.umkc.edu/lizhu
Z. Li: ECE 5582 Computer Vision, 2020 p.1
slides created with WPS Office Linux and EqualX LaTex equation editor
![Page 2: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/2.jpg)
Outline
Recap: Eigenface Fisherface HW-3: Fisherface and Laplacian face
Graph Embedding Laplacianface Graph Fourier Transform
Course project topics
Summary
p.2Z. Li: ECE 5582 Computer Vision, 2020
![Page 3: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/3.jpg)
Subspace Learning for Face Recognition
Project face images to a subspace with basis A Matlab: x=faces*A(:,1:kd)
eigf1
eigf2
eigf3
= 10.9* + 0.4* + 4.7*
p.3Z. Li: ECE 5582 Computer Vision, 2020
![Page 4: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/4.jpg)
PCA & Fisher’s Linear Discriminant
• Between-class scatter
• Within-class scatter
• Where– c is the number of classes– i is the mean of class i
– | i | is number of samples of i..
Tii
c
iiBS ))((
1 --=å
=
å å= Î
--=c
i x
TikikW
ik
xS1
))((
1
2
1 2
p.4Z. Li: ECE 5582 Computer Vision, 2020
![Page 5: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/5.jpg)
Eigen vs Fisher Projection
p.5
• PCA (Eigenfaces)
Maximizes projected total scatter
• Fisher’s Linear Discriminant
Maximizes ratio of projected between-class to projected within-class scatter, solved by the generalized Eigen problem:
WSWW TT
WPCA maxarg=
WSW
WSWW
WT
BT
Wfld maxarg=
1 2
PCA
Fisher ���= ����
Z. Li: ECE 5582 Computer Vision, 2020
![Page 6: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/6.jpg)
LDA implementation myLDA.m
p.6
% compute class meanmx = mean(x); ids = unique(y); m = length(ids);Sb = zeros(kd, kd); for k=1:m indx = find(y==ids(k)); nk(k) = length(indx); % class mean m_cx(k,:) = mean(x(indx, :)); % between class scatter Sb = Sb + nk(k)*(m_cx(k,:) - mx)'*(m_cx(k,:) - mx);end
% compute intra-class scatterSw = zeros(kd, kd); for k=1:m indx = find(y==ids(k)); nk(k) = length(indx); % remove mean xk = x(indx, :) - repmat(m_cx(k,:), [nk(k), 1]); % adding up Sw = Sw + (xk'*xk);end
Z. Li: ECE 5582 Computer Vision, 2020
![Page 7: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/7.jpg)
LDA Implementation
Find projection by generalized Eigen problem solution
p.7
% generalized eigen problem[A, v]=eigs(Sb, Sw); if dbg figure(31); subplot(2,2,1); imagesc(Sb); colormap('gray'); title('S_b'); subplot(2,2,2); imagesc(Sw); colormap('gray'); title('S_w'); z = x*A; dist = pdist2(z, z); subplot(2,2,3); imagesc(dist); title('dist(j,k)'); subplot(2,2,4); stem(diag(v), '.'); grid on; hold on; title('eig v');end
���= ����
Z. Li: ECE 5582 Computer Vision, 2020
![Page 8: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/8.jpg)
Fisherface Basis
It is interesting to compare Fisherface with Eigenface basis
p.8
FisherfaceEigenface
Z. Li: ECE 5582 Computer Vision, 2020
![Page 9: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/9.jpg)
Fisherface Performance
Fisher vs Eigenface performance: (fisherface.m) 1200 face images, 144 subjects Eigen kd=32
p.9
144 subjectsROC
Z. Li: ECE 5582 Computer Vision, 2020
![Page 10: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/10.jpg)
Outline
Recap: Eigenface Fisherface
Graph Embedding Locality Preserving Projection Laplacian Face Graph Fourier Transform
Summary
p.10Z. Li: ECE 5582 Computer Vision, 2020
![Page 11: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/11.jpg)
Locality Preserving Projection
Recall the dimension reduction formulations: find w, s.t y=wx: PCA:
LDA:
p.11
max�
wTSw
S = SB + SW
Z. Li: ECE 5582 Computer Vision, 2020
![Page 12: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/12.jpg)
LPP Formulation – Affinity
To preserve local affinity relationship Affinity map (Similar to the Laplacian Eigen Map)
Selection of heat kernel size and threshold are important Hint: affinity matrix should be sparse
p.12
affinity histogram
Z. Li: ECE 5582 Computer Vision, 2020
![Page 13: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/13.jpg)
LPP Formulation – Affinity Supervised
How to utilize the label info ? Heat map mapping is good for intra-class affinity modelling,
but how about intra-class affinity ? One direct solution is to set affinity to zero for intra class
pairs
p.13
% LPP - compute affinityf_dist1 = pdist2(x1, x1);% heat kernel sizemdist = mean(f_dist1(:)); h = -log(0.15)/mdist; S1 = exp(-h*f_dist1); id_dist = pdist2(ids, ids);subplot(2,2,3); imagesc(id_dist); title('label distance');S2=S1; S2(find(id_dist~=0)) = 0; subplot(2,2,4); imagesc(S1); colormap('gray'); title('affinity-supervised');
Z. Li: ECE 5582 Computer Vision, 2020
![Page 14: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/14.jpg)
LPP- Affinity Preserving Projection
Find a projection that best preserves the affinity matrix
p.14
min�
����������� −���
� , �. �. � =�X
min�
����������X� −�X��
�
�. �, ���� = �−exp��X� −X��
��ℎ , �X �X� −X�� ≤ �
0, �X��
Z. Li: ECE 5582 Computer Vision, 2020
![Page 15: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/15.jpg)
LPP Formulation
L = D-S: nxn matrix, called graph Laplacian
Normalizing factor: nxn D Diagonal matrix, entry Dii = sum of col/row affinity The larger the value, the more important data point is
Constraint on D:
p.15Z. Li: ECE 5582 Computer Vision, 2020
![Page 16: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/16.jpg)
Generalized Eigen Problem
Now the formulation is,
Lagranian
by KKT (Karush-Khun-Tucker) Condition, it is solved by a generalized Eigen problem
p.16
���, �� = ������� − �����X��−1�
Z. Li: ECE 5582 Computer Vision, 2020
X He, S Yan, Y Hu, P Niyogi, HJ Zhang, “Face Recognition Using Laplacianface”, IEEE Trans PAMI, vol. 27 (3), 328-340, 2005.
![Page 17: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/17.jpg)
Matlab Implementation
laplacianface.m
p.17
%LPPn_face = 1200; n_subj = length(unique(ids(1:n_face)));
% eigenface kd=32; x1 = faces(1:n_face,:)*A1(:,1:kd); ids=ids(1:n_face);
% LPP - compute affinityf_dist1 = pdist2(x1, x1);% heat kernel sizemdist = mean(f_dist1(:)); h = -log(0.15)/mdist; S1 = exp(-h*f_dist1); figure(32); subplot(2,2,1); imagesc(f_dist1); colormap('gray'); title('d(x_i, d_j)');subplot(2,2,2); imagesc(S1); colormap('gray'); title('affinity'); %subplot(2,2,3); grid on; hold on; [h_aff, v_aff]=hist(S(:), 40); plot(v_aff, h_aff, '.-'); % utilize supervised infoid_dist = pdist2(ids, ids);subplot(2,2,3); imagesc(id_dist); title('label distance');S2=S1; S2(find(id_dist~=0)) = 0; subplot(2,2,4); imagesc(S1); colormap('gray'); title('affinity-supervised');
% laplacian facelpp_opt.PCARatio = 1; [A2, eigv2]=LPP(S2, lpp_opt, x1);
Z. Li: ECE 5582 Computer Vision, 2020
![Page 18: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/18.jpg)
Laplacian Face
Now, we can model face as a LPP projection:
p.18
Eigenface Laplacian Face
Z. Li: ECE 5582 Computer Vision, 2020
![Page 19: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/19.jpg)
Laplacian vs Eigenface
1200 faces, 144 subjects
p.19Z. Li: ECE 5582 Computer Vision, 2020
![Page 20: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/20.jpg)
LPP and PCA Graph Embedding is an unifying theory on dimension
reduction PCA becomes special case of LPP, if we do not enforce local
affinity
p.20Z. Li: ECE 5582 Computer Vision, 2020
![Page 21: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/21.jpg)
LPP and LDA
How about LDA ? Recall within class scatter:
p.21
This is i-th classData covariance
Li has diagonal entry of 1/ni,Equal affinity among data points
Z. Li: ECE 5582 Computer Vision, 2020
![Page 22: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/22.jpg)
LPP and LDA
Now consider the between class scatter C is the data covariance,
regardless of label L is graph Laplacian computed
from the affinity rule that,
p.22
���, �� = �1��
, �X X�, X��X� XX G �X��� �
0, �X��
Z. Li: ECE 5582 Computer Vision, 2020
![Page 23: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/23.jpg)
LDA as a special case of LPP
The same generalized Eigen problem
p.23Z. Li: ECE 5582 Computer Vision, 2020
![Page 24: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/24.jpg)
Graph Embedding Interpretation of PCA/LDA/LPP
Affinity graph S, determines the embedding subspace W, via
PCA and LDA are special cases of Graph Embedding PCA:
LDA
LPP
p.24
���� = �−exp��X� −X��
��ℎ , �X �X� −X�� ≤ �
0, �X��
���� = �1��
, �X X�, X� ∈ ��
0, �X��
���� = 1/�
Z. Li: ECE 5582 Computer Vision, 2020
![Page 25: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/25.jpg)
Applications: facial expression embedding
Facial expressions embedded in a 2-d space via LPP
p.25
frown
sad
happy
neutral
Z. Li: ECE 5582 Computer Vision, 2020
![Page 26: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/26.jpg)
Application: Compression of SIFT
Compression of SIFT, preserve matching relationship, rather than reconstruction:
Z. Li: ECE 5582 Computer Vision, 2020 p.26
� =argmin�
����������X� −�X��
�
![Page 27: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/27.jpg)
Homework-3: Subspace Methods
Objective: Understand the graph embedding connections among popular
subspace methods like PCA, LDA and LPP Practical experiences with serious size data set
Data Set: https://umkc.app.box.com/s/0qu7tc3jb88at2h53l1dpcuqkt9pn
7ww 417 subjects, 6650 image face data set, pre-processed to
20x20 pel images, intensity normalized to [0, 1] Add your own face images, 10~15, frontal
Tasks: Compute Eigenface, Fisherface and Laplacianface models ROC plot on verification performance mAP for retrieval/identification performance
Z. Li: ECE 5582 Computer Vision, 2020 p.27
![Page 28: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/28.jpg)
HW-3 test run
Laplacian face is powerful.
Z. Li: ECE 5582 Computer Vision, 2020 p.28
![Page 29: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/29.jpg)
Graph Fourier Transform David I. Shuman, Sunil K. Narang, Pascal Frossard, Antonio Ortega, Pierre Vandergheynst:
The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains. IEEE Signal Process. Mag. 30(3): 83-98 (2013)
Z. Li: ECE 5582 Computer Vision, 2020 p.29
![Page 30: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/30.jpg)
Signal on Graph
non-uniformly sampled
p.30Z. Li: ECE 5582 Computer Vision, 2020
![Page 31: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/31.jpg)
Graph Fourier Transform
GFT is different from Laplacian Embedding:
p.31Z. Li: ECE 5582 Computer Vision, 2020
![Page 32: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/32.jpg)
GFT Example
Graph Laplacian
p.32Z. Li: ECE 5582 Computer Vision, 2020
![Page 33: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/33.jpg)
Normalized Graph Laplacian
Normalize by edge pair degree
p.33Z. Li: ECE 5582 Computer Vision, 2020
![Page 34: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/34.jpg)
Graph Frourier Transform
Analogous to FT
p.34Z. Li: ECE 5582 Computer Vision, 2020
![Page 35: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/35.jpg)
Graph Spectrum
Freq interpretation
p.35Z. Li: ECE 5582 Computer Vision, 2020
![Page 36: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/36.jpg)
GetGFT.m
Implementation of GFT: affinity graph sparsity control is important
Z. Li: ECE 5582 Computer Vision, 2020 p.36
![Page 37: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/37.jpg)
Graph Signal Smoothness
Quadratic form on L:
p.37Z. Li: ECE 5582 Computer Vision, 2020
most smooth: low freq dominating
![Page 38: Lec 15: Graph Embedding & Laplacianface · PCA is a special case of graph embedding oFully connected affinity map, equal importance LDA is a special case of graph embedding oFully](https://reader034.vdocument.in/reader034/viewer/2022050222/5f6791215ef52e15e842027f/html5/thumbnails/38.jpg)
Summary
Graph Laplacian Embedding is an unifying theory for feature space dimension reduction PCA is a special case of graph embedding
o Fully connected affinity map, equal importance LDA is a special case of graph embedding
o Fully connected intra classo Zero affinity inter class
LPP: preserves pair wise affinity. GFT: eigen vectors of graph Laplacian, has Fourier Transform
like characteristics. Many applications in Face recognition Pose estimation Facial expression modeling Compression of Graph signals.
p.38Z. Li: ECE 5582 Computer Vision, 2020