lecture1 jps

8/2/2019 Lecture1 Jps

1/100

Factor and Component

Analysis

esp. Principal Component

Analysis (PCA)


2/100

We study phenomena that can not be directly observed ego, personality, intelligence in psychology

Underlying factors that govern the observed data

We want to identify and operate with underlying latentfactors rather than the observed data

E.g. topics in news articles

Transcription factors in genomics

We want to discover and exploit hidden relationships

beautiful car and gorgeous automobile are closely related

So are driver and automobile

But does your search engine know this?

Reduces noise and error in results

Why Factor or Component Analysis?


3/100

We have too many observations and dimensions To reason about or obtain insights from

To visualize

Too much noise in the data

Need to reduce them to a smaller set of factors Better representation of data without losing much information

Can build more effective data analyses on the reduced-

dimensional space: classification, clustering, pattern recognition

Combinations of observed variables may be more effective

bases for insights, even if physical meaning is obscure

Why Factor or Component Analysis?


4/100

Discover a new set of factors/dimensions/axes againstwhich to represent, describe or evaluate the data

For more effective reasoning, insights, or better visualization

Reduce noise in the data

Typically a smaller set of factors: dimension reduction Better representation of data without losing much information

Can build more effective data analyses on the reduced-

dimensional space: classification, clustering, pattern recognition

Factors are combinations of observed variables

May be more effective bases for insights, even if physical

meaning is obscure

Observed data are described in terms of these factors rather than

in terms of original variables/dimensions

Factor or Component Analysis


5/100

Basic Concept

Areas of variance in data are where items can be best discriminatedand key underlying phenomena observed Areas of greatest signal in the data

If two items or dimensions are highly correlated or dependent They are likely to represent highly related phenomena

If they tell us about the same underlying variance in the data, combiningthem to form a single measure is reasonable

Parsimony

Reduction in Error

So we want to combine related variables, and focus on uncorrelatedor independent ones, especially those along which the observationshave high variance

We want a smaller set of variables that explain most of the variance

in the original data, in more compact and insightful form


6/100

Basic Concept What if the dependences and correlations are not so

strong or direct?

And suppose you have 3 variables, or 4, or 5, or 10000?

Look for the phenomena underlying the observedcovariance/co-dependence in a set of variables Once again, phenomena that are uncorrelated or independent,

and especially those along which the data show high variance

These phenomena are called factors or principalcomponents or independent components, depending onthe methods used Factor analysis: based on variance/covariance/correlation

Independent Component Analysis: based on independence


7/100

Principal Component Analysis

Most common form of factor analysis

The new variables/dimensions

Are linear combinations of the original ones

Are uncorrelated with one another

Orthogonal in original dimension space

Capture as much of the original variance inthe data as possible

Are called Principal Components


8/100

Some Simple Demos

http://www.cs.mcgill.ca/~sqrt/dimr/dimreducti
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


9/100

What are the new axes?

Original Variable A

OriginalVariableB

PC 1PC 2

Orthogonal directions of greatest variance in data Projections along PC1 discriminate the data most along

any one axis
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


10/100

Principal Components

First principal component is the direction of

greatest variability (covariance) in the data

Second is the next orthogonal (uncorrelated)direction of greatest variability

So first remove all the variability along the first

component, and then find the next direction of

greatest variability

And so on
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


11/100

Principal Components

Analysis (PCA) Principle

Linear projection method to reduce the number of parameters

Transfer a set of correlated variables into a new set of uncorrelatedvariables

Map the data into a space of lower dimensionality Form of unsupervised learning

Properties It can be viewed as a rotation of the existing axes to new positions in the

space defined by original variables

New axes are orthogonal and represent the directions with maximumvariability
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


12/100

Computing the Components Data points are vectors in a multidimensional space

Projection of vectorx onto an axis (dimension) u is u.x

Direction of greatest variability is that in which the averagesquare of the projection is greatest I.e. u such that E((u.x)2) over all x is maximized

(we subtract the mean along each dimension, and center the

original axis system at the centroid of all data points, for simplicity) This direction ofu is the direction of the first Principal Component
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


13/100

Computing the Components

E((u.x)2

) = E ((u.x) (u.x)T

) = E (u.x.xT

.uT

) The matrix C = x.xT contains the correlations

(similarities) of the original axes based on how the

data values project onto them

So we are looking for w that maximizes uCuT, subject

to u being unit-length

It is maximized when w is the principal eigenvector of

the matrix C, in which case uCuT = uuT = ifu is unit-length, where is the

principal eigenvalue of the correlation matrix C The eigenvalue denotes the amount of variability captured

along that dimension
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


14/100

Why the Eigenvectors?

Maximise uTxxTu s.tuTu = 1

Construct Langrangian uTxxTuuTu

Vector of partial derivatives set to zero

xxTuu =(xxTI) u = 0As u 0 then u must be an eigenvector ofxxT with

eigenvalue
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


15/100

Singular Value Decomposition

The first root is called the prinicipal eigenvalue which has anassociated orthonormal (uTu = 1) eigenvector u

Subsequent roots are ordered such that 1> 2 > > M with

rank(D) non-zero values.

Eigenvectors form an orthonormal basis i.e. uiTuj = ij

The eigenvalue decomposition ofxxT = UUT

where U = [u1, u2, , uM] and = diag[1, 2, , M]

Similarly the eigenvalue decomposition ofxTx = VVT

The SVD is closely related to the above x=U 1/2 VT

The left eigenvectors U, right eigenvectors V,

singular values = square root of eigenvalues.
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


16/100

Computing the Components Similarly for the next axis, etc.

So, the new axes are the eigenvectors of the matrix ofcorrelations of the original variables, which capturesthe similarities of the original variables based on howdata samples project to them

Geometrically: centering followed by rotation

Linear transformation
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


17/100

PCs, Variance and Least-Squares

The first PC retains the greatest amount of variation in the

sample The kth PC retains the kth greatest fraction of the variation in the

sample The kth largest eigenvalue of the correlation matrix C is the

variance in the sample along the kth PC

The least-squares view: PCs are a series of linear least squares

fits to a sample, each orthogonal to all previous ones
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


18/100

How Many PCs?

For n original dimensions, correlation

matrix is nxn, and has up to n

eigenvectors. So n PCs. Where does dimensionality reduction

come from?
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


19/100

0

5

10

15

20

25

PC1 PC2 PC3 PC4 PC5 PC6 PC7 PC8 PC9 PC10

Dimensionality Reduction

Can ignore the components of lesser significance.

You do lose some information, but if the eigenvalues aresmall, you dont lose much n dimensions in original data

calculate n eigenvectors and eigenvalues

choose only the first p eigenvectors, based on their eigenvalues

final data set has only p dimensions
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


20/100

Eigenvectors of a Correlation

Matrix
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


21/100

Computing and Using PCA:

Text Search Example
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


22/100

PCA/FA

Principal Components Analysis Extracts all the factor underlying a set of variables The number of factors = the number of variables

Completely explains the variance in each variable

Parsimony?!?!?!

Factor Analysis Also known as Principal Axis Analysis

Analyses only the shared variance

NO ERROR!
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


23/100

Vector Space Model for Documents

Documents are vectors in multi-dim Euclidean space Each term is a dimension in the space: LOTS of them

Coordinate of doc d in dimension t is prop. to TF(d,t) TF(d,t) is term frequency of t in document d

Term-document matrix, with entries being normalizedTF values t*d matrix, for t terms and d documents

Queries are also vectors in the same space

Treated just like documents

Rank documents based on similarity of their vectorswith the query vector

Using cosine distance of vectors
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


24/100

Problems

Looks for literal term matches Terms in queries (esp short ones) dont always

capture users information need well

Problems: Synonymy: other words with the same meaning

Car and automobile

Polysemy: the same word having other meanings Apple (fruit and company)

What if we could match against concepts,that represent related words, rather thanwords themselves
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


25/100

Example of Problems

-- Relevant docs may not have the query terms

but may have many related terms

-- Irrelevant docs may have the query terms

but may not have any related terms


26/100

Latent Semantic Indexing (LSI)

Uses statistically derived conceptual indicesinstead of individual words for retrieval

Assumes that there is some underlying orlatent

structure in word usage that is obscured byvariability in word choice

Key idea: instead of representing documents andqueries as vectors in a t-dim space of terms

Represent them (and terms themselves) as vectors in alower-dimensional space whose axes are concepts thateffectively group together similar words

These axes are the Principal Components from PCA
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


27/100

Example

Suppose we have keywords Car, automobile, driver, elephant

We want queries on car to also get docs about drivers andautomobiles, but not about elephants

What if we could discover that the cars, automobiles and driversaxes are strongly correlated, but elephants is not

How? Via correlations observed through documents

If docs A & B dont share any words with each other, but bothshare lots of words with doc C, then A & B will be consideredsimilar

E.g A has cars and drivers, B has automobiles and drivers

When you scrunch down dimensions, small differences(noise) gets glossed over, and you get desired behavior
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


28/100

LSI and Our Earlier Example

-- Relevant docs may not have the query terms

but may have many related terms

-- Irrelevant docs may have the query terms

but may not have any related terms


29/100

LSI: Satisfying a query

Take the vector representation of the query in theoriginal term space and transform it to new space Turns out dq = xq

t T -1

places the query pseudo-doc at the centroid of its

corresponding terms locations in the new space

Similarity with existing docs computed by taking dotproducts with the rows of the matrix D2Dt
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


30/100

Computing and Using LSI

(PCA)

Definitions

Let t be the total number of index terms

Let N be the number of documents

Let (Mij) be a term-document matrix with t rows

and N columns

Entries are TF-based weights
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


31/100

Latent Semantic Indexing

The matrix (Mij) can be decomposed into 3

matrices (SVD) as follows: (Mij) = (U) (S) (V)t

(U) is the matrix of eigenvectors derived from (M)(M)t

(V)t is the matrix of eigenvectors derived from (M)t(M)

(S) is an r x r diagonal matrix of singular values

r = min(t,N) that is, the rank of (Mij)

Singular values are the positive square roots of the eigen

values of (M)(M)t (also (M)t(M))


32/100

Example

term ch2 ch3 ch4 ch5 ch6 ch7 ch8 ch9

controllability 1 1 0 0 1 0 0 1

observability 1 0 0 0 1 1 0 1

realization 1 0 1 0 1 0 1 0

feedback 0 1 0 0 0 1 0 0

controller 0 1 0 0 1 1 0 0

observer 0 1 1 0 1 1 0 0

transfer

function0 0 0 0 1 1 0 0

polynomial 0 0 0 0 1 0 1 0

matrices 0 0 0 0 1 0 1 1

U (9x7) =

0.3996 -0.1037 0.5606 -0.3717 -0.3919 -0.3482 0.1029

0.4180 -0.0641 0.4878 0.1566 0.5771 0.1981 -0.1094

0.3464 -0.4422 -0.3997 -0.5142 0.2787 0.0102 -0.2857

0.1888 0.4615 0.0049 -0.0279 -0.2087 0.4193 -0.6629

0.3602 0.3776 -0.0914 0.1596 -0.2045 -0.3701 -0.1023

0.4075 0.3622 -0.3657 -0.2684 -0.0174 0.2711 0.5676

0.2750 0.1667 -0.1303 0.4376 0.3844 -0.3066 0.1230

0.2259 -0.3096 -0.3579 0.3127 -0.2406 -0.3122 -0.2611

0.2958 -0.4232 0.0277 0.4305 -0.3800 0.5114 0.2010

S (7x7) =

3.9901 0 0 0 0 0 00 2.2813 0 0 0 0 0

0 0 1.6705 0 0 0 0

0 0 0 1.3522 0 0 0

0 0 0 0 1.1818 0 0

0 0 0 0 0 0.6623 0

0 0 0 0 0 0 0.6487

V (7x8) =

0.2917 -0.2674 0.3883 -0.5393 0.3926 -0.2112 -0.4505

0.3399 0.4811 0.0649 -0.3760 -0.6959 -0.0421 -0.1462

0.1889 -0.0351 -0.4582 -0.5788 0.2211 0.4247 0.4346

-0.0000 -0.0000 -0.0000 -0.0000 0.0000 -0.0000 0.0000

0.6838 -0.1913 -0.1609 0.2535 0.0050 -0.5229 0.3636

0.4134 0.5716 -0.0566 0.3383 0.4493 0.3198 -0.2839

0.2176 -0.5151 -0.4369 0.1694 -0.2893 0.3161 -0.5330

0.2791 -0.2591 0.6442 0.1593 -0.1648 0.5455 0.2998This happens to be a rank-7 matrix

-so only 7 dimensions required

Singular values = Sqrt of Eigen values of AAT

T
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


33/100

Dimension Reduction in LSI

The key idea is to map documents and

queries into a lower dimensional space (i.e.,

composed of higher level concepts which are

in fewer number than the index terms)

Retrieval in this reduced concept space might

be superior to retrieval in the space of index

terms
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


34/100

Dimension Reduction in LSI

In matrix (S), select only k largest values

Keep corresponding columns in (U) and (V)t

The resultant matrix (M)k is given by

(M)k = (U)k (S)k (V)tk

where k, k < r, is the dimensionality of the concept space

The parameter k should be

large enough to allow fitting the characteristics of the data

small enough to filter out the non-relevant representationaldetail


35/100

PCs can be viewed as Topics

In the sense of having to find quantities that are not observable directly

Similarly, transcription factors in biology, as unobservable causal bridges

between experimental conditions and gene expression
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


36/100

Computing and Using LSI

=

=

mxn

A

mxr

U

rxr

D

rxn

VT

Terms

Documents

=

=

mxn

k

mxk

Uk

kxk

Dk

kxn

VT

k

Terms

Documents

Singular Value

Decomposition

(SVD):

Convert term-document

matrix into 3matrices

U, S and V

Reduce Dimensionality:

Throw out low-order

rows and columns

Recreate Matrix:

Multiply to produceapproximate term-

document matrix.

Use new matrix to

process queries

OR, better, map query to

reduced space

M U S Vt Uk Sk Vkt
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


37/100

Following the Example





feedback 0 1 0 0 0 1 0 0

controller 0 1 0 0 1 1 0 0

observer 0 1 1 0 1 1 0 0

transfer

function0 0 0 0 1 1 0 0

polynomial 0 0 0 0 1 0 1 0

matrices 0 0 0 0 1 0 1 1

U (9x7) =

0.3996 -0.1037 0.5606 -0.3717 -0.3919 -0.3482 0.1029

0.4180 -0.0641 0.4878 0.1566 0.5771 0.1981 -0.1094

0.3464 -0.4422 -0.3997 -0.5142 0.2787 0.0102 -0.2857

0.1888 0.4615 0.0049 -0.0279 -0.2087 0.4193 -0.6629

0.3602 0.3776 -0.0914 0.1596 -0.2045 -0.3701 -0.1023

0.4075 0.3622 -0.3657 -0.2684 -0.0174 0.2711 0.5676

0.2750 0.1667 -0.1303 0.4376 0.3844 -0.3066 0.1230

0.2259 -0.3096 -0.3579 0.3127 -0.2406 -0.3122 -0.2611

0.2958 -0.4232 0.0277 0.4305 -0.3800 0.5114 0.2010

S (7x7) =

3.9901 0 0 0 0 0 00 2.2813 0 0 0 0 0

0 0 1.6705 0 0 0 0

0 0 0 1.3522 0 0 0

0 0 0 0 1.1818 0 0

0 0 0 0 0 0.6623 0

0 0 0 0 0 0 0.6487

V (7x8) =

0.2917 -0.2674 0.3883 -0.5393 0.3926 -0.2112 -0.4505

0.3399 0.4811 0.0649 -0.3760 -0.6959 -0.0421 -0.1462

0.1889 -0.0351 -0.4582 -0.5788 0.2211 0.4247 0.4346

-0.0000 -0.0000 -0.0000 -0.0000 0.0000 -0.0000 0.0000

0.6838 -0.1913 -0.1609 0.2535 0.0050 -0.5229 0.3636

0.4134 0.5716 -0.0566 0.3383 0.4493 0.3198 -0.2839

0.2176 -0.5151 -0.4369 0.1694 -0.2893 0.3161 -0.5330

0.2791 -0.2591 0.6442 0.1593 -0.1648 0.5455 0.2998This happens to be a rank-7 matrix

-so only 7 dimensions required

Singular values = Sqrt of Eigen values of AAT

T

F ll hi ill b h k k (2)
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


38/100

U2 (9x2) =

0.3996 -0.1037

0.4180 -0.0641

0.3464 -0.4422

0.1888 0.4615

0.3602 0.3776

0.4075 0.3622

0.2750 0.1667

0.2259 -0.3096

0.2958 -0.4232

S2 (2x2) =

3.9901 0

0 2.2813

V2 (8x2) =

0.2917 -0.2674

0.3399 0.4811

0.1889 -0.0351-0.0000 -0.0000

0.6838 -0.1913

0.4134 0.5716

0.2176 -0.5151

0.2791 -0.2591

U (9x7) =

0.3996 -0.1037 0.5606 -0.3717 -0.3919 -0.3482 0.1029

0.4180 -0.0641 0.4878 0.1566 0.5771 0.1981 -0.1094

0.3464 -0.4422 -0.3997 -0.5142 0.2787 0.0102 -0.2857

0.1888 0.4615 0.0049 -0.0279 -0.2087 0.4193 -0.6629

0.3602 0.3776 -0.0914 0.1596 -0.2045 -0.3701 -0.1023

0.4075 0.3622 -0.3657 -0.2684 -0.0174 0.2711 0.5676

0.2750 0.1667 -0.1303 0.4376 0.3844 -0.3066 0.1230

0.2259 -0.3096 -0.3579 0.3127 -0.2406 -0.3122 -0.2611

0.2958 -0.4232 0.0277 0.4305 -0.3800 0.5114 0.2010

S (7x7) =

3.9901 0 0 0 0 0 0

0 2.2813 0 0 0 0 00 0 1.6705 0 0 0 0

0 0 0 1.3522 0 0 0

0 0 0 0 1.1818 0 0

0 0 0 0 0 0.6623 0

0 0 0 0 0 0 0.6487

V (7x8) =

0.2917 -0.2674 0.3883 -0.5393 0.3926 -0.2112 -0.4505

0.3399 0.4811 0.0649 -0.3760 -0.6959 -0.0421 -0.1462

0.1889 -0.0351 -0.4582 -0.5788 0.2211 0.4247 0.4346-0.0000 -0.0000 -0.0000 -0.0000 0.0000 -0.0000 0.0000

0.6838 -0.1913 -0.1609 0.2535 0.0050 -0.5229 0.3636

0.4134 0.5716 -0.0566 0.3383 0.4493 0.3198 -0.2839

0.2176 -0.5151 -0.4369 0.1694 -0.2893 0.3161 -0.5330

0.2791 -0.2591 0.6442 0.1593 -0.1648 0.5455 0.2998

U2*S2*V2 will be a 9x8 matrix

That approximates original matrix

T

Formally, this will be the rank-k (2)

matrix that is closest to M in the

matrix norm sense
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


39/100





feedback 0 1 0 0 0 1 0 0

controller 0 1 0 0 1 1 0 0

observer 0 1 1 0 1 1 0 0

transfer

function0 0 0 0 1 1 0 0

polynomial 0 0 0 0 1 0 1 0

matrices 0 0 0 0 1 0 1 1





feedback 0 1 0 0 0 1 0 0

controller 0 1 0 0 1 1 0 0

observer 0 1 1 0 1 1 0 0

transfer

function0 0 0 0 1 1 0 0

polynomial 0 0 0 0 1 0 1 0

matrices 0 0 0 0 1 0 1 1

termterm ch2ch2 ch3ch3 ch4ch4 ch5ch5 ch6ch6 ch7ch7 ch8ch8 ch9ch9

controllabilitycontrollability 11 11 00 00 11 00 00 11

observabilityobservability 11 00 00 00 11 11 00 11

realizationrealizat ion 11 00 11 00 11 00 11 00

feedbackfeedback 00 11 00 00 00 11 00 00

controllercontroller 00 11 00 00 11 11 00 00

observerobserver 00 11 11 00 11 11 00 00

transfer

function

transfer

function00 00 00 00 11 11 00 00

polynomial polynomial 00 00 00 00 11 00 11 00

matricesmatrices 00 00 00 00 11 00 11 11

K=2

K=6

One component ignored

5 components ignored

U6S6V6T

U2S2V2T

USVT

0.52835834 0.42813724 0.30949408 0.0 1.1355368 0.5239192 0.46880865 0.5063048

0.5256176 0.49655432 0.3201918 0.0 1.1684579 0.6059082 0.4382505 0.50338876

0.6729299 -0.015529543 0.29650056 0.0 1.1381099 -0.0052356124 0.82038856 0.6471

-0.0617774 0.76256883 0.10535021 0.0 0.3137232 0.9132189 -0.37838274 -0.06253

0.18889774 0.90294445 0.24125765 0.0 0.81799114 1.0865396 -0.1309748 0.17793834

0.25334513 0.95019233 0.27814224 0.0 0.9537667 1.1444798 -0.071810216 0.2397161

0.21838559 0.55592346 0.19392742 0.0 0.6775683 0.6709899 0.042878807 0.2077163

0.4517898 -0.033422917 0.19505836 0.0 0.75146574 -0.031091988 0.55994695 0.4345

0.60244554 -0.06330189 0.25684044 0.0 0.99175954 -0.06392482 0.75412846 0.5795

1.0299273 1.0099105 -0.029033005 0.0 0.9757162 0.019038305 0.035608776 0.98004794

0.96788234 -0.010319378 0.030770123 0.0 1.0258299 0.9798115 -0.03772955 1.0212346

0.9165214 -0.026921304 1.0805727 0.0 1.0673982 -0.052518982 0.9011715 0.055653755

-0.19373542 0.9372319 0.1868434 0.0 0.15639876 0.87798584 -0.22921464 0.12886547

-0.029890355 0.9903935 0.028769515 0.0 1.0242295 0.98121595 -0.03527296 0.020075336

0.16586632 1.0537577 0.8398298 0.0 0.8660687 1.1044582 0.19631699 -0.11030859

0.035988174 0.01172187 -0.03462495 0.0 0.9710446 1.0226605 0.04260301 -0.023878671

-0.07636017 -0.024632007 0.07358454 0.0 1.0615499 -0.048087567 0.909685 0.050844945

0.05863098 0.019081593 -0.056740552 0.0 0.95253044 0.03693092 1.0695065 0.96087193

1.1630535 0.67789733 0.17131016 0.0 0.85744447 0.30088043 -0.025483057 1.0295205

0.7278324 0.46981966 -0.1757451 0.0 1.0910251 0.6314231 0.11810507 1.0620605

0.78863835 0.20257005 1.0048805 0.0 1.0692837 -0.20266426 0.9943222 0.106248446

-0.03825318 0.7772852 0.12343567 0.0 0.30284256 0.89999276 -0.3883498 -0.06326774

0.013223715 0.8118903 0.18630582 0.0 0.8972661 1.1681904 -0.027708884 0.11395822

0.21186034 1.0470067 0.76812166 0.0 0.960058 1.0562774 0.1336124 -0.2116417

-0.18525022 0.31930918 -0.048827052 0.0 0.8625925 0.8834896 0.23821498 0.1617572

-0.008397698 -0.23121 0.2242676 0.0 0.9548515 0.14579195 0.89278513 0.1167786

0.30647483 -0.27917668 -0.101294056 0.0 1.1318822 0.13038804 0.83252335 0.70210195

U4S4V4T

K=4

=U7S7V7T

3 components

ignored

What should be the value of k?
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.html


40/100

Querying

To query forfeedback controller, the query vector would be

q = [0 0 0 1 1 0 0 0 0]' (' indicates

transpose),

Let q be the query vector. Then the document-space vector

corresponding to q is given by:

q'*U2*inv(S2) = Dq

Point at the centroid of the query terms poisitions in thenew space.

For thefeedback controllerquery vector, the result is:

Dq = 0.1376 0.3678

To find the best document match, we compare theDq vector

against all the document vectors in the 2-dimensionalV2

space. The document vector that is nearest in direction to

Dq is the best match. The cosine values for the eight

document vectors and the query vector are:

-0.3747 0.9671 0.1735 -0.9413 0.0851 0.9642 -0.7265 -0.3805





feedback 0 1 0 0 0 1 0 0

controller 0 1 0 0 1 1 0 0

observer 0 1 1 0 1 1 0 0

transfer

function0 0 0 0 1 1 0 0

polynomial 0 0 0 0 1 0 1 0

matrices 0 0 0 0 1 0 1 1





feedback 0 1 0 0 0 1 0 0

controller 0 1 0 0 1 1 0 0

observer 0 1 1 0 1 1 0 0

transfer

function0 0 0 0 1 1 0 0

polynomial 0 0 0 0 1 0 1 0

matrices 0 0 0 0 1 0 1 1

termterm ch2ch2 ch3ch3 ch4ch4 ch5ch5 ch6ch6 ch7ch7 ch8ch8 ch9ch9

controllabilitycontrollability 11 11 00 00 11 00 00 11

observabilityobservability 11 00 00 00 11 11 00 11

realizationrealization 11 00 11 00 11 00 11 00

feedbackfeedback 00 11 00 00 00 11 00 00

controllercontroller 00 11 00 00 11 11 00 00

observerobserver 00 11 11 00 11 11 00 00

transfer

function

transfer

function00 00 00 00 11 11 00 00

polynomial polynomial 00 00 00 00 11 00 11 00

matricesmatrices 00 00 00 00 11 00 11 11

-0.37 0.967 0.173 -0.94 0.08 0.96 -0.72 -0.38

U2 (9x2) =

0.3996 -0.1037

0.4180 -0.0641

0.3464 -0.4422

0.1888 0.4615

0.3602 0.3776

0.4075 0.3622

0.2750 0.1667

0.2259 -0.3096

0.2958 -0.4232

S2 (2x2) =

3.9901 0

0 2.2813

V2 (8x2) =

0.2917 -0.2674

0.3399 0.4811

0.1889 -0.0351

-0.0000 -0.0000

0.6838 -0.1913

0.4134 0.5716

0.2176 -0.5151

0.2791 -0.2591
http://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.mcgill.ca/~sqrt/dimr/dimreduction.htmlhttp://www.cs.