Upload File

pca & fisher discriminant analysisjavierhr/files/slidespca.pdf · pca vs lda 23 pca: perform...

  • Home
  • Documents
  • PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional
28
Dimensionality Reduction Lecturer: Javier Hernandez Rivera 30 th September 2010 MAS 622J/1.126J: Pattern Recognition & Analysis "A man's mind, stretched by new ideas, may never return to it's original dimensions" Oliver Wendell Holmes Jr.

Upload: others

Post on 02-Oct-2020

9 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Dimensionality Reduction

Lecturer: Javier Hernandez Rivera

30th September 2010

MAS 622J/1.126J: Pattern Recognition & Analysis

"A man's mind, stretched by new ideas, may never return to it's original

dimensions" Oliver Wendell Holmes Jr.

Page 2: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Outline

2

Before

Linear Algebra

Probability

Likelihood Ratio

ROC

ML/MAP

Today

Accuracy, Dimensions & Overfitting (DHS 3.7)

Principal Component Analysis (DHS 3.8.1)

Fisher Linear Discriminant/LDA (DHS 3.8.2)

Other Component Analysis Algorithms

Page 3: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Accuracy, Dimensions & Overfitting

3

Classification accuracy depends upon the dimensionality?

Training

Testing

O(c) < O(log n) < O(n) < O(n log n) < O(n^2) < O(n^k) < O(2^n) < O(n!) < O(n^n)

Complexity (computational/temporal)

Big O notation

if

Page 4: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Overfitting

4

ClassificationRegression

Error

Complexity

Overfitting

Page 5: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Dimensionality Reduction

5

Optimal data representation

2D: Evaluating students’ performance

# As

# study hours

# As

# study hours

Unnecessary complexity

Page 6: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Optimal data representation

3D: Find the most informative point of view

Dimensionality Reduction

6

Page 7: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Principal Component Analysis

7

Assumptions for new basis:

Large variance has important structure

Linear projection

Orthogonal basis

Y =WTX

d dim, n samples

dim i of sample j

X 2Rd;nxij

Y 2Rk;nWT 2Rk;d

projected data

k basis of d dimwTj wi = 1 i = j wTj wi = 0 i 6= j

Page 8: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Optimal W?

1. Minimize projection cost (Pearson, 1901)

2. Maximize projected variance (Hotelling, 1933)

Principal Component Analysis

8

minkx¡ x̂k

Page 9: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Principal Component Analysis

9

Covariance Algorithm

1. Zero mean:

2. Unit variance:

3. Covariance matrix:

4. Compute eigenvectors of ∑: WT

5. Project X onto the k principal components

xij = xij ¡¹i ¹i =1n

Pn

j=1 xij

xij =xij¾j

¾i =1

n¡1Pn

j=1(xij ¡ ¹i)2

§ =XXT

Ykxn

WT

kxd

Xdxn

= x

k < d

Page 10: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Principal Component Analysis

10

How many PCs?

There are d eigenvectors

Choose first p eigenvectors based on their eigenvalues

Final data has only k dimensions

14% of Energy

Normalized eigenvalue of 3rd PC

Pk

i=1¸iP

n

i=1¸i

> Threshold (0.9 or 0.95)

Y =WXkxn kxd dxn

Page 11: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

PCA Applications: EigenFaces

11

Images from faces:

• Aligned

• Gray scale

• Same size

• Same lighting

Xdxn

d = # pixels/image

n = # images

Wdxd

eigenFaces

Sirovich and Kirby, 1987

Matthew Turk and Alex Pentland, 1991

eig vect (X XT)

Page 12: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

PCA Applications: EigenFaces

12

Faces EigenFaces

Eigenfaces are standardized face ingredients

A human face may be considered to be a combination of these standard faces

Page 13: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

PCA Applications: EigenFaces

13

x

xW

Dimensionality Reduction

ynewClassification

Outlier Detection

argminckynew ¡ yck < µ1

argmaxckycar ¡ yck > µ2

ycar

Page 14: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

PCA Applications: EigenFaces

14

Reconstructed images

(adding 8 PC at each step)

= y1 + y2 + yk+ …

Reconstructionx w1 w2 wk

Reconstructing with

partial information

Page 15: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Principal Component Analysis

15

Any problems?

Let’s define a samples as an image of100x100 pixels

d = 10,000 X XT is 10,000 x 10,000

O(d2d3)

computing eigenvectors becomes infeasible

Possible solution (just if n<<d)Use WT = (XE)T

where E є Rn,k are k eigenvectors of XTX

100 px

100 p

x

Page 16: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

SVD Algorithm

1. Zero Mean:

2. Unit Variance:

3. Compute SVD of X:

4. Projection matrix:

5. Project data

Principal Component Analysis

16

xij = xij ¡¹i ¹i =1n

Pn

j=1 xij

xij =xij¾j

¾i =1

n¡1Pn

j=1(xij ¡ ¹i)2

X =U§VT

W=U

Why W = U?

Ykxn

Wkxd

Xdxn

= x

k <= n < d

Page 17: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Principal Component Analysis

17

MATLAB: [U S V] = svd(A);

Data

Columns

are data

points

Right Singular

Vectors

Columns are

eigenvectors of

XXT

Left Singular

Vectors

Rows are eigenvectors

of XTX

Singular

Values

Diagonal matrix

of sorted values

Xdxn

Udxd

Sdxn

= x VT

nxn

x

Page 18: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

PCA Applications: Action Unit Recognition

18

T. Simon, J. Hernandez, 2009

Page 19: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Applications

19

Visualize high dimensional data (e.g. AnalogySpace)

Find hidden relationships (e.g. topics in articles)

Compress information

Avoid redundancy

Reduce algorithm complexity

Outlier detection

Denoising

Demos:

http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html

Page 20: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

PCA for pattern recognition

20

• higher variance

• bad for discriminability

• smaller variance

• good discriminability

Principal Component Analysis

Fisher Linear Discriminant

Linear Discriminant Analysis

Page 21: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Linear Discriminant Analysis

21

Assumptions for new basis:

Maximize distance between projected class means

Minimize projected class variance

y = wTx

Page 22: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Algorithm

1. Compute class means

2. Compute

3. Project data

Linear Discriminant Analysis

22

Objective

w = S¡1W (m2 ¡m1)

argmaxwJ(w) =wTSBwwTSWw

SW =P2

j

Px2Cj(x¡mj)(x¡mj)

T

SB = (m2¡m1)(m2¡m1)T

mi =1ni

Px2Ci x

y = wTx

Page 23: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

PCA vs LDA

23

PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional space as possible.

LDA: Perform dimensionality reduction while preserving as much of the class discriminatory information as possible.

Page 24: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Other Component Analysis Algorithms

24

Independent Component Analysis (ICA)

PCA: uncorrelated PCsICA: independent PCs

Blind source separation

M. Poh, D. McDuff,

and R. Picard 2010

Recall:

independent uncorrelated

uncorrelated X independent

Page 25: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Independent Component Analysis

25

Sound

Source 1

Sound

Source 2

Mixture 1

Mixture 2

Output 1

Output 2

I

C

A

Sound

Source 3

Mixture 3

Output 3

Page 26: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Other Component Analysis Algorithms

26

Curvilinear Component Analysis (CCA)

PCA/LDA: linear projection

CCA: non-linear projection

while preserving proximity

between points

Page 27: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

What you should know?

27

Why dimensionality reduction?

PCA/LDA assumptions

PCA/LDA algorithms

PCA vs LDA

Applications

Possible extensions

Page 28: PCA & Fisher Discriminant Analysisjavierhr/files/slidesPCA.pdf · PCA vs LDA 23 PCA: Perform dimensionality reduction while preserving as much of the variance in the high dimensional

Recommended Readings

28

Tutorials A tutorial on PCA. J. Shlens

A tutorial on PCA. L. Smith

Fisher Linear Discriminat Analysis. M. Welling

Eigenfaces (http://www.cs.princeton.edu/~cdecoro/eigenfaces/)

C. DeCoro

Publications Eigenfaces for Recognition. M. Turk and A. Pentland

Using Discriminant Eigenfeatures for Image Retrieval.

D. Swets, J. Weng

Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. P. Belhumeur, J. Hespanha, and D. Kriegman

PCA versus LDA. A. Martinez, A. Kak

http://www.cs.princeton.edu/~cdecoro/eigenfaces/

MLCC 2015 Dimensionality Reduction and PCAlcsl.mit.edu/courses/mlcc/mlcc2015/slides/MLCC_05_PCA.pdf · MLCC 2015 Dimensionality Reduction and PCA Lorenzo Rosasco UNIGE-MIT-IIT June

Dimensionality Reduction of Hyperspectral Image Using ...h.web.umkc.edu/hangr/papers/Dimensionality Reduction of Hyperspe… · Using Spatial Regularized Local Graph Discriminant

Lecture 8 - Computer Science - Western University · Lecture 8. Today Continue with Dimensionality Reduction Last lecture: PCA This lecture: Fisher Linear Discriminant. PCA finds

Dimensionality Reduction: Linear Discriminant Analysis and ... · Linear Discriminant Analysis (LDA, LDiscA) and Principal Component Analysis (PCA) Summarize D-dimensional input data

Dimensionality Reduction and Principle Components Analysiswcohen/10-601/601-pca.pdf · • Principle Components Analysis (PCA) – Example (Bishop, ch 12) – PCA vs linear regression

PCA & Fisher Discriminant Analysiscourses.media.mit.edu/2010fall/mas622j/ProblemSets/slidesPCA.pdf · Recommended Readings 28 Tutorials A tutorial on PCA. J. Shlens A tutorial on

Lecture 10. PCA, SVD and Fisher Linear Discriminantayuille/courses/Stat161-261-Spring14/RevisedLectureNote10.pdfFigure 6: This type of data is bad for PCA. Fisher’s Linear Discriminant

CSE446: Dimensionality Reduction and PCA Spring 2017 · 2017-05-25 · Spring 2017 Slides adapted from Carlos Guestrin and Luke Zettlemoyer. Dimensionality reduction • Input data

Using PCA and Factor Analysis for Dimensionality Reduction

DATA MINING LECTURE 7 Dimensionality Reduction PCA – SVD (Thanks to Jure Leskovec, Evimaria Terzi)

Linear discriminant analysis, two classes Linear ...Gutierrez-Osuna... · Linear discriminant analysis, two-classes • Objective –LDA seeks to reduce dimensionality while preserving

Linear & Non-Linear Discriminant Analysis · Discriminant Analysis of Child Faces •Does face shape reveal whether child is male or female?! •Used 45 child faces! •PCA to reduce

Dimensionality reduction (pca)

DIMENSIONALITY REDUCTION: PCA, MDSrita/uml_course/lectures/PCA_MDS.pdf · Relation to PCA PCA MDS Spectral Decomposition Covariance matrix ( D x D) Gram matrix (n x n) Eigenvalues

CSE 446 Dimensionality Reduction and PCA Winter 2012

MULTIVARIATE (& MULTIMODALITY) IMAGE ANALYSIS - … · A more complex case ... PCA decorrelates multivariate data, finds useful components, reduces dimensionality. • PCA is only

Dimensionality Reduction PCA · 2017. 2. 24. · 1 Dimensionality Reduction PCA Machine Learning –CSE446 David Wadden (slides provided by Carlos Guestrin) University of Washington

LEC 3: Fisher Discriminant Analysis (FDA) - SJSUgchen/Math285S16/lec3fda.pdf · LEC 3: Fisher Discriminant Analysis (FDA) – A Supervised Dimensionality Reduction Approach Dr. Guangliang

SVD and PCA - Western Michigan Universityalfuqaha/summer14/cs6530/... · SVD and PCA. Real data usually have thousands , or millions of dimensions E.g., web documents, where the dimensionality

Lecture 8Lecture 8 Today Continue with Dimensionality Reduction Last lecture: PCA This lecture: Fisher Linear Discriminant PCA finds the most accurate data representation in a lower

Principle Component Analysis (PCA) Networks (§ 5.8) PCA: a statistical procedure –Reduce dimensionality of input vectors Too many features, some of them

Linear Discriminant Analysis(LDA)syllabus.cs.manchester.ac.uk/pgt/COMP61021/lectures/LDA.pdf• Case Study: PCA vs. LDA • Relevant Issues ... • Linear discriminant analysis (LDA)

Lecture 10. PCA, SVD and Fisher Linear Discriminant

· PDF fileLECTURE 10: Linear Discriminant Analysis • Linear Discriminant Analysis, two classes • Linear Discriminant Analysis, C classes • I-DA vs. PCA example

DATA MINING LECTURE 8 Dimensionality Reduction PCA -- SVD

1586 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE …DR, independent component analysis (ICA), maximum noise fraction (MNF), PCA-DR, principal components analysis (PCA), virtual dimensionality

Linear discriminant analysis, two classes Linear ...gn.dronacharya.info/CSEDept/Downloads/Question... · LDA vs. PCA • Limitations of LDA • Variants of LDA • Other dimensionality

Locality Adaptive Discriminant Analysis - GitHub Pages Locality Adaptive... · Linear Discriminant Analysis (LDA) is a popu-lar technique for supervised dimensionality reduc-tion,

PCA, Kernel PCA, ICAninamf/courses/601sp15/slides/22_pca-04-09-20… · PCA, Kernel PCA, ICA Learning Representations. Dimensionality Reduction. • High-Dimensions = Lot of Features

Facial PCA and Fisher Discriminant Analysis

Optimally Weighted PCA for High-Dimensional Heteroscedastic … · 2018. 11. 26. · 1. Introduction. Principal Component Analysis (PCA) is a classical method for reducing the dimensionality

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE ... · Dimensionality reduction. PCA has been extensively used to reduce the dimensionality of SIFT vectors [20], [6]. In this way,

LEC 4: Discriminant Analysis for Classificationgchen/Math285S16/lec4lda.pdf · Last time: FDA (dimensionality reduction) ... Matlab/Python commands. Discriminant Analysis for Classification

Week 4, Lecture 7 - Dimensionality Reduction: PCA and NMF · Week 4, Lecture 7 - Dimensionality Reduction: PCA and NMF Author: Aaron Meyer Created Date: 3/3/2020 4:38:52 PM

Local Fisher Discriminant Analysis for Supervised Dimensionality Reduction