Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj

Go With The FlowA New Manifold Modeling and Learning Framework for Image Ensembles

Aswin C. SankaranarayananRice University

Sensor Data Deluge

Concise Models• Efficient processing / compression requires

concise representation

• Sparsity of an individual image

pixels largewaveletcoefficients(blue = 0)

Concise Models• Efficient processing / compression requires

concise representation• Our interest in this talk: Collections of images

Concise Models• Our interest in this talk:

Collections of image parameterized by q \in

Q– translations of an object

q: x-offset and y-offset

– rotations of a 3D objectq: pitch, roll, yaw

– wedgeletsq: orientation and offset

Concise Models• Our interest in this talk:

Collections of image parameterized by q \in

Q– translations of an object

q: x-offset and y-offset

– rotations of a 3D objectq: pitch, roll, yaw

– wedgeletsq: orientation and offset

• Image articulation manifold

Image Articulation Manifold• N-pixel images:

• K-dimensional articulation space

• Thenis a K-dimensional manifoldin the ambient space

• Very concise model

articulation parameter space

Smooth IAMs• N-pixel images:

• Local isometry: image distance parameter space distance

• Linear tangent spacesare close approximationlocally

articulation parameter space

Smooth IAMs• N-pixel images:

• Local isometry: image distance parameter space distance

• Linear tangent spacesare close approximationlocally

articulation parameter space

Ex: Manifold Learning

LLEISOMAPLEHEDiff. Geo …

• K=1rotation

Ex: Manifold Learning

• K=2rotation and scale

Theory/Practice Disconnect: Smoothness

• Practical image manifolds are not smooth!

• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]

Tangent approximations ?Isometry ?

Theory/Practice Disconnect: Smoothness

• Practical image manifolds are not smooth!

• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]

Tangent approximations ?Isometry ?

Failure of Tangent Plane Approx.

• Ex: cross-fading when synthesizing / interpolating images that should lie on manifold

Input Image Input Image

Geodesic Linear path

Failure of Local Isometry

• Ex: translation manifold

all blue imagesare equidistantfrom the red image

• Local isometry

– satisfied only when sampling is dense

0 20 40 60 80 1000

0.5

1

1.5

2

2.5

3

3.5

Translation q in [px]

Euc

lidea

n di

stan

ce

Tools for manifold processing

Geodesics, exponential maps, log-maps, Riemannian metrics, Karcher means, …

Smooth diff manifold

Algebraic manifolds Data manifolds

LLE, kNN graphs

Point cloud model

The concept ofTransport operators

))(()( xfIxIrefqq

Beyond point cloud model for image manifolds

refIfI qq

Example

Example: Translation• 2D Translation manifold

barring boundary related issues

• Set of all transport operators =• Beyond a point cloud model

– Action of the articulation is more accurate and meaningful

1I

11101 )( ),()( xxfxIxI0I

2R

Optical Flow

• Generalizing this idea: Pixel correspondances

• Idea:OF between two images is a natural and accurate transport operator

(Figures from Ce Liu’s optical flow page)

OFfrom I1 to I2

I1 and I2

Optical Flow Transport

0q

1q 2q

IAM

2qI1q

I0qI

Articulations

• Consider a reference imageand a K-dimensional articulation

• Collect optical flows fromto all images reachable by aK-dimensional articulation

Optical Flow Transport

0q

1q 2q

IAM

OFM at 0qI

2qI1q

I0qI

Articulations

• Consider a reference imageand a K-dimensional articulation

• Collect optical flows fromto all images reachable by aK-dimensional articulation

• Theorem: Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth)

OFM is Smooth (Rotation)

00q 30q 60q 30q 60q

-80 -60 -40 -20 0 20 40 60 800

50

100

150

200

Articulation q in [ ]

Inte

nsity

-80 -60 -40 -20 0 20 40 60 80-20

-10

0

10

20

Articulation q in [ ]

Opt

ical

flow

vx i

n pi

xels

Articulation θ in [⁰]

Inte

nsity

I(θ)

Op. fl

ow v

(θ)

Pixel intensityat 3 points

Flow (nearly linear)

Main results• Local model at each

• Each point on the OFM defines a transport operator– Each transport operator maps

to one of its neighbors

• For a large class of articulations, OFMs are smooth and locally isometric– Traditional manifold processing

techniques work on OFMs

0q

1q 2q

IAM

OFM at 0qI

2qI1q

I0qI

Articulations

0qI

0qI

Linking it all together

0q

1q 2q

IAM

OFM at 0qI

2qI1q

I0qI

Articulations

0e

1e 2e

Nonlinear dim. reduction

The non-differentiablity does not dissappear --- it is embedded in the mapping from OFM to the IAM.

However, this is a known map

))(()( such that },{ xfIxIIfI refref qqqq

The Story So Far…

0q

1q 2q

IAM

OFM at 0qI

2qI1q

I0qI

Articulations

Tangent space at 0qI

Articulations

0q

1q 2q

2qI1q

I0qI IAM

Input Image Input Image

Geodesic Linear pathIAM

OFM

OFM Synthesis

ISOMAP embedding error for OFM and IAM

2D rotations

Reference image

Manifold Learning

Manifold Learning

-6 -4 -2 0 2 4 6-5

-4

-3

-2

-1

0

1

2

3

4

5Two-dimensional Isomap embedding (with neighborhood graph).

Embedding of OFM

2D rotations

Reference image

Data196 images of two bears moving linearlyand independently

IAM OFM

TaskFind low-dimensional embedding

OFM Manifold Learning

IAM OFM

Data196 images of a cup moving on a plane

Task 1Find low-dimensional embedding

Task 2Parameter estimation for new images(tracing an “R”)

OFM ML + Parameter Estimation

• Point on the manifold such that the sum of geodesic distances to every other point is minimized

• Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al]

Karcher Mean

10 images from an IAM

ground truth KM OFM KM linear KM

Manifold Charting• Goal: build a generative model for an entire

IAM/OFM based on a small number of base images

• Ex: cube rotating about axis. All cube images can be representing using 4 reference images + OFMs

• Many applications– selection of target templates for classification– “next-view” selection for adaptive sensing applications

Greedy charting

Optimal charting

0q 90q 180q 180q 90q

IAM

Summary• IAMs a useful concise model for many image

processing problems involving image collections and multiple sensors/viewpoints

• But practical IAMs are non-differentiable– IAM-based algorithms have not lived up to their promise

• Optical flow manifolds (OFMs)– smooth even when IAM is not– OFM ~ nonlinear tangent space– support accurate image synthesis, learning, charting, …

• Barely discussed here: OF enables the safe extension of differential geometry concepts– Log/Exp maps, Karcher mean, parallel transport, …

Open Questions

• Our treatment is specific to image manifolds under

brightness constancy

• What are the natural transport operators for other data manifolds?

dsp.rice.edu

Related Work• Analytic transport operators

– transport operator has group structure [Xiao and Rao 07][Culpepper and Olshausen 09] [Miller and Younes 01] [Tuzel et al 08]

– non-linear analytics [Dollar et al 06]– spatio-temporal manifolds [Li and Chellappa 10]– shape manifolds [Klassen et al 04]

• Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups)

• In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations

Limitations• Brightness constancy

– Optical flow is no longer meaningful

• Occlusion– Undefined pixel flow in theory, arbitrary flow estimates in

practice– Heuristics to deal with it

• Changing backgrounds etc.– Transport operator assumption too strict– Sparse correspondences ?

Open Questions• Theorem:

random measurements stably embed aK-dim manifoldwhp [B, Wakin, FOCM ’08]

• Q: Is there an analogousresult for OFMs?

Image Articulation Manifold

• Linear tangent space at is K-dimensional – provides a mechanism to

transport along manifold

– problem: since manifold is non-differentiable, tangent approximation is poor

• Our goal: replace tangent spacewith new transport operator that respects the nonlinearity of the imaging process

Tangent space at 0qI

Articulations

0q

1q 2q

2qI1q

I0qI IAM

OFM Implementation details

Reference Image

Pairwise distances and embedding

-5 0 5-4

-2

0

2

4

100 200 300 400

100

200

300

400

0 2 4 6 8 100

0.1

0.2

0.3

0.4

0.5

Res

idua

l var

ianc

e

Isomap dimensionality

-5 0 5-4

-2

0

2

4100 200 300 400

100

200

300

400

0 2 4 6 8 100

0.1

0.2

0.3

0.4

0.5

Res

idua

l var

ianc

e

Isomap dimensionality

Flow Embedding

Occlusion• Detect occlusion using forward-backward flow

reasoning

• Remove occluded pixel computations

• Heuristic --- formal occlusion handling is hard

Occluded

History of Optical Flow

• Dark ages (<1985)– special cases solved– LBC an under-determined set of linear equations

• Horn and Schunk (1985) – Regularization term: smoothness prior on the flow

• Brox et al (2005)– shows that linearization of brightness constancy (BC) is

a bad assumption– develops optimization framework to handle BC directly

• Brox et al (2010), Black et al (2010), Liu et al (2010)– practical systems with reliable code