local multidimensional scaling [1em] for nonlinear ......for measuring local continuity examples...

Local MultidimensionalScaling

For Nonlinear DimensionReduction, Graph Layoutand Proximity Analysis

L. Chen

Motivation

Some Existing Methods

New proposal: LMDS

The LC Meta-criterionfor Measuring LocalContinuity

Examples

Graph-layout and Energyfunctions

Generalized LMDS

Conclusion andDiscussion

Local Multidimensional Scaling

For Nonlinear Dimension Reduction, GraphLayout and Proximity Analysis

Lisha Chen (Statistics, Yale)Joint work with Andreas Buja (Upenn)

Feb 6th, 2007



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Machine Learning Facial Images

I Facial images vary from moment to moment.

I The mystery of perception:How does brain perceive constancy in flux?

I Goal: Construct machines to capture the variability of facialimages.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Characterizing Image Variability

High-dimensional data . . .

I An image is regarded as a collection of numbers.

I Each number is the light intensity at an image pixel.

I An Example: 64 by 64 images ⇒ 4096-D data.

Low-dimensional manifold . . .

Degree of freedom

I Left-right pose

I Up-down pose

I Lighting-direction

I Expression change(not shown)



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Classical Method 1: PCA

I Principal Components: Fit the best ellipsoid to multivariatedata x1, ..., xN ∈ IRp.

I Linear method: Provide a sequence of best linearapproximation to the data.

I Uses: Plot the wide dimensions; interpret their directions.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Classical Method 2: Multidimensional Scaling

Multidimensional Scaling(MDS) for proximity data . . .

I Make a map from given pairwise dissimilarities.

I A Toy Example:

D =

0 3 43 0 54 5 0

MDS for dimension reduction . . .

I Calculate dissimilarities(Dij) in high-dimensional space.

I Match Dij in low-dimensional space.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Nonlinear Structure in High Dimensional Data

I Classical methods fail to find nonlinear structure.

−200 0 200 400

−200

0

200

400

Swiss Roll

−300 −200 −100 0 100 200 300

−300

−200

−100

0

100

200

300

2D PCA

0 500 1000 1500

0

500

1000

1500

True 2D Manifold

I Nonlinear dimension reduction methods:I Isomap(Tenenbaum et. al., 2000)I LLE(Roweis & Saul, 2000)I Hessian Eigenmaps(Donoho & Carrie, 2003)I Laplacian Eigenmaps (Belkin & Niyogi, 2003)I Diffusion Maps(Coifman et. al., 2005)



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Isometric Feature Mapping (Isomap)

The idea . . .

I Characterize the intrinsic geometry by geodesicdistances(shortest path).

I Match geodesic distances by Euclidean distances inlow-dimensional space.

The algorithm . . .

1 : Construct neighborhoodgraph.

2 : Compute shortest path.

3 : Construct embedding by

MDS.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Local Linear Embedding(LLE)

The idea . . .

I Nearby points lie on a locally linear patch.

I Characterize the local geometry by linear coefficientsthat reconstruct each data point from its neighbors.

The algorithm . . .

1 : Select neighbors.

2 : Construct weights.

ε(W ) =∑

i

|Xi−∑

j∈N(i)

WijXj |2

3 : Embedding.

Φ(Y ) =∑

i

|Yi−∑

j∈N(i)

WijYj |2



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Commonality of The Proposals: Localization

I Discard the information of large distances.

I Use local information:I Fixed-radius metric neighborhoods.I K nearest neighborhoods(KNN).

I Find a way to reassemble local structure and flatten themanifold.

I Our goal: Apply localization to MDS.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Revisit MDS

I Data:Objects i = 1, 2, ...,N (subjects, stimuli, graph nodes,...)Dissimilarities Dij for all pairs of objects i and j

I Goal: Visualization of objectsFind configuration {xi ∈ IRk |i = 1..N} such that:

‖xi − xj‖ ≈ Dij

k: embedding dimension

I Stress function: Goodness of fit criterionThe simplest version:

MDSD(x1, ..., xN) =∑

i,j=1..N

(Dij − ||xi − xj ||)2



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Localization Problem in MDS

A simple proposal . . .

I Localization: Drop the large dissimilarities.

I Stress function becomes:

MDSD(x1, ..., xN) =∑

(i,j)∈E

(Dij − ||xi − xj ||)2

E = edge set of local links

Unstable system: Graef and Spence (1979)



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


New Proposal: Local MDS(LMDS)

The idea . . .

I Approximate the small dissimilarities.

I Give a proper amount of repulsive force amongthe points to avoid instability.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS



I Step 1: Localization: E = edge set of local links; EC =edge set of non-local links.

I Step 2: Replace large distances (EC ) with ∞, but withinfinitesimal weight.

SD(x1, ..., xN) =∑

(ij)∈E

(Dij − ‖xi − xj‖)2

+ w ·∑

(ij)∈EC

(D∞ − ‖xi − xj‖)2

I Step 3: Take limit w → 0 and D∞ →∞ subject to2 w D∞ = t.

LMDSD(x1, ..., xN) =∑

(ij)∈E

(Dij − ‖xi − xj‖)2

− t ·∑

(ij)∈EC

‖xi − xj‖



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS



LMDSD(x1, ..., xN) =∑

(ij)∈E

(Dij − ‖xi − xj‖)2

− t ·∑

(ij)∈EC

‖xi − xj‖

I The first term: Local stress to match small dissimilarities.

I The second term: Repulsive force for spreading out.

I The choice of t: t = τ · (K/N) ·medianE (Dij)

I τ repulsion tuning parameter, usually 0.001 ∼ 1.I K : the number of neighbors for each point .



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


The LC Meta-criterion: Motivation andDefinition

I Motivation

I Notation:I ND

K (i): Data point i ’s K nearest neighbors with regard tothe target dissimilarities Dij .

I N XK (i): Data point i ’s K nearest neighbors with regard to

the configuration distances ‖xi − xj‖.I | S |: Cardinality of a set S.

I The local version of the LC meta-criterion:

NK (i) = | NDK (i) ∩ NX

K (i) |

I The global version of the LC meta-criterion:

NK =1

N

N∑i=1

NK (i)



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


The LC meta-criterion: Normalization andAdjustment for Baselines

I Compare LC meta-criteria for different values of K.

I Normalization:

MK =1

KNK

I The problem of meta-criterion for large K : MN−1 = 1.

I Adjusted LC Meta-criterion:

MadjK = MK −

K

N − 1



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Example 1: Sculpture Face

−50 0 50

−40

−30

−20

−10

0

10

20

30

3D LMDS

Left−right Pose

Up−

dow

n P

ose

−50 0 50

−40

−30

−20

−10

0

10

20

30

Left−right Pose

Ligh

ting

Dire

ctio

n



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS



Adjust the repulsion tuning parameter τ in LMDS:

5e−04 5e−03 5e−02 5e−01

2.5

3.0

3.5

4.0

4.5

5.0

K=6

τ

KM

K



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Example 1: Sculpture FaceMethods PCA MDS Isomap LLE LMDS

N6 2.6 3.1 4.5 2.8 5.2

−15 −5 0 5 10

−15

−5

515

−10 0 5 10

−10

05

15

−10 0 5 10

−10

05

15

−2 −1 0 1 2

−2

−1

01

2

−2 −1 0 1 2

−2

−1

01

2

−2 −1 0 1 2

−2

−1

01

2

−2 −1 0 1 2−

20

12

−1.0 0.0 1.0

−1.

00.

01.

0

−1.0 0.0 1.0

−1.

00.

01.

0−0.25 −0.15 −0.05 0.05−

0.25

−0.

100.

00−0.25 −0.15 −0.05 0.05

−0.

25−

0.10

0.00

−0.25 −0.15 −0.05 0.05

−0.

25−

0.10

0.00

−50 0 50

−50

050

−40 −20 0 20

−40

−20

020

−40 −20 0 20−

40−

200

20

Left−

right

Pos

eU

p−do

wn

Pos

eLi

ghtin

g D

irect

ion

PCA MDS ISOMAP LLE LMDS



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS



−15 −10 −5 0 5 10 15

−15

−10

−5

05

1015

3D PCA (M6=2.6)

−2 −1 0 1 2

−2

−1

01

2

3D ISOMAP (M6=4.5)

−0.05 0.00 0.05

−0.

050.

000.

05

3D LLE (M6=2.8)

−50 0 50

−50

050

3D LMDS (M6=5.2)



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS



5 10 20 50 100 200 500

0.2

0.4

0.6

0.8

1.0

K

MK

MK

AdjustedMK



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS



5 10 20 50 100 200 500

0.0

0.2

0.4

0.6

0.8

1.0

K

MK

Ko=4Ko=6Ko=8Ko=10Ko=25Ko=125Ko=650

5 10 20 50 100 200 500

0.0

0.2

0.4

0.6

0.8

1.0

K

Adj

uste

d M

K

Ko=4Ko=6Ko=8Ko=10Ko=25Ko=125Ko=650



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Example 2: Frey Face

−600 −400 −200 0 200 400 600

−60

0−

400

−20

00

200

400

600 3D PCA

Left−right Pose

Ser

ious

−S

mili

ng

−2 −1 0 1 2

−2

−1

01

2

3D MDS

Left−right Pose

Ser

ious

−S

mili

ng

−2 −1 0 1 2

−2

−1

01

2

3D ISOMAP

Left−right Pose

Ser

ious

−S

mili

ng

−0.10 −0.05 0.00 0.05

−0.

10−

0.05

0.00

0.05

3D LLE

Left−right Pose

Ser

ious

−S

mili

ng



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS



−10000 −5000 0 5000 10000

−10

000

−50

000

5000

1000

0

3D LMDS K0=12

Left−right Pose

Ser

ious

−S

mili

ng

−3e+06 −1e+06 1e+06 2e+06 3e+06

−3e

+06

−1e

+06

1e+

063e

+06

3D LMDS K0=4

Left−right Pose

Ser

ious

−S

mili

ng



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Example 2: Frey Face3D PCA

Left−right Pose

Ser

ious

−S

mili

ng

3D MDS

Left−right Pose

Ser

ious

−S

mili

ng

3D ISOMAP

Left−right Pose

Ser

ious

−S

mili

ng

3D LLE

Left−right Pose

Ser

ious

−S

mili

ng

3D LMDS K0=12

Left−right Pose

Ser

ious

−S

mili

ng

3D LMDS K0=4

Left−right Pose

Ser

ious

−S

mili

ng



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Example 2: Frey Face3D PCA (M12=3.6)

Left−right Pose

Ser

ious

−S

mili

ng

3D MDS (M12=4.8)

Left−right Pose

Ser

ious

−S

mili

ng

3D ISOMAP (M12=4.2)

Left−right Pose

Ser

ious

−S

mili

ng

3D LLE (M12=3.2)

Left−right Pose

Ser

ious

−S

mili

ng

3D LMDS K0=12 (M12=4.6)

Left−right Pose

Ser

ious

−S

mili

ng

3D LMDS K0=4 (M12=5.1)

Left−right Pose

Ser

ious

−S

mili

ng



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS



5 10 20 50 100

0.3

0.4

0.5

0.6

K

MK

Ko=4Ko=8Ko=12Ko=16Ko=20Ko=50Ko=100Ko=150Ko=1964

5 10 20 50 100

−0.

10−

0.05

0.00

0.05

K

Adj

uste

d M

K w

ith M

DS



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Outline for the Rest of the Talk

I Graphs and energy-based models.

I Stress in LMDS and energy in graph layout.

I Clustering postulate.

I Generalized LMDS: A Box-Cox family of energy functions.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Graph-layout

I A graph G = (V ,E ):I V ↔ Entities: Telephone numbers, carbon atoms . . .I E ↔ Relations: Phone calls, chemical bonds . . .

I Goal: Two-dimensional layout ↔ Visualize the relationalinformation.

I Algorithm: Force-directedPlacement(Eades, 1987)

I A physical system:Vertices → Steel rings;Edges → Springs.

I Best layout is achieved atthe minimal energy state.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Energy-based Models

I Notation:I E = edge set; EC = non-edge set; V 2 = E ∪ EC .I Denote dij = ‖xi − xj‖ = drawn distances.I U: Energy of the system.I f (·): energy corresponding to attraction

g(·):energy corresponding to repulsion

I Common in graph layout: attraction-repulsion forces

U =∑E

f (dij) −∑V 2

g(dij)

I LMDS and energy-based models.

LMDSD =∑E

(Dij − dij)2 − t ·

∑EC

dij



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Energy-based Models

Some energy functions . . .

I Eades (1984)

U =∑

E

dij ·(log(dij)− 1− dij

−2)

+∑V 2

dij−1

I Fruchterman & Reingold (1991)

U =∑

E

dij3/3 −

∑V 2

log(dij)

I Davidson & Harel (1996)

U =∑

E

dij2 +

∑V 2

dij−2

I Noack (2003), LinLog model

U =∑

E

dij −∑V 2

log(dij)



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Clustering Postulate

I A graph with two clusters:

n1ne

n2

I Problem: What should be the inter-cluster distance d?

I Energy:U(d) = ne f (d)− n1n2g(d)

I Clustering Postulate(Noack 2004):I U(d) should be minimized at d = n1n2/ne = 1/C .I C = ne/n1n2 is called coupling strength, C < 1.

I Stationary condition yields:

f ′(d) = d · g ′(d)

I Weak Clustering Postulate: More strongly coupled clustersshould be placed more closely to each other.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Distance-weighted Graphs

I Assume target distances Dij for (i , j) ∈ E .

I Extended clustering condition: In the 2-cluster situation,require the minimum at

d = D · ( 1C)λ

I D: the target distanceI ( 1

C)λ: a generalized clustering factor.

I λ > 0: clustering power; C < 1: coupling strength.

I New stationarity condition:

f ′(d) =

(d

D

) 1λ

g ′(d)

I f (d) and g(d) satisfy the extended coupling condition:

f (d) =dµ+ 1

λ − 1

µ + 1λ

, g(d) =dµ − 1

µD

1λ

with logarithmic fill-in for zero exponents.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


A Box-Cox Family of Energy Functions

I Energy:

U ∼∑

E

(dij

µ+ 1λ −1

µ+ 1λ

− Dij1λ

dijµ−1µ

)− t

∑E c

dijµ−1µ

I Special cases:I LMDS: λ = µ = 1I Energy for unweighted graph(Dij = 1):

I Noack(LinLog): λ = 1, µ = 0I Fruchterman&Reingold: λ = 1

3, µ = 0

I Davison&Harel: λ = 14, µ = −2

Advantages . . .

I Flexibility for achieving desirable configurations.

I Clustering power λ: visualize the clusters.



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Example 3: Olivetti Faces



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Example 3: Olivetti FacesMethods λ = 0.25 λ = 0.5 λ = 1.0 λ = 1.5

N4 0.64 1.01 1.44 1.57

−6000 −2000 0 2000 6000

−60

00−

2000

020

0060

00

λ=0.25

−15000 −5000 0 5000 15000−15

000

−50

000

5000

1500

0 λ=0.5

−20000 0 10000

−20

000

010

000

λ=1

−40000 −20000 0 20000 40000

−40

000

020

000

4000

0 λ=1.5



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Conclusion and Discussion

I Nonlinear dimension reduction, proximity analysis andgraph-layout

I ContiguityI DistancesI Localization

I Graph Layout: Two Fundamental ApproachesI Complete the graph (Isomap, Krustal and Seery(1980))I Apply repulsive force

I B-C Energy Functions

I The Selection Problem: LC Meta-Criteria



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Diagnostic Plot: Frey Face

500 1000 1500

05000

10000

15000

20000

Diagnostic Plot

D_ij

d_ij



L. Chen

Motivation


New proposal: LMDS


Examples


Generalized LMDS


Diagnostic Plot: Frey Face

local multidimensional scaling [1em] for nonlinear ......for measuring local continuity examples...

Documents