![Page 1: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/1.jpg)
Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj
Manifold Learning in the WildA New Manifold Modeling and Learning Framework for Image Ensembles
Aswin C. SankaranarayananRice University
![Page 2: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/2.jpg)
Sensor Data Deluge
![Page 3: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/3.jpg)
Concise Models• Efficient processing / compression requires
concise representation• Our interest in this talk: Collections of images
![Page 4: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/4.jpg)
Concise Models• Our interest in this talk:
Collections of image parameterized by q \in
Q– translations of an object
q: x-offset and y-offset
– rotations of a 3D objectq: pitch, roll, yaw
– wedgeletsq: orientation and offset
![Page 5: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/5.jpg)
Concise Models• Our interest in this talk:
Collections of image parameterized by q \in
Q– translations of an object
q: x-offset and y-offset
– rotations of a 3D objectq: pitch, roll, yaw
– wedgeletsq: orientation and offset
• Image articulation manifold
![Page 6: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/6.jpg)
Image Articulation Manifold• N-pixel images:
• K-dimensional articulation space
• Thenis a K-dimensional manifoldin the ambient space
• Very concise model
articulation parameter space
![Page 7: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/7.jpg)
Smooth IAMs• N-pixel images:
• Local isometry image distance parameter space distance
• Linear tangent spacesare close approximationlocally
• Low dimensional articulation space
articulation parameter space
![Page 8: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/8.jpg)
Smooth IAMs
articulation parameter space
• N-pixel images:
• Local isometry image distance parameter space distance
• Linear tangent spacesare close approximationlocally
• Low dimensional articulation space
![Page 9: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/9.jpg)
Smooth IAMs
articulation parameter space
• N-pixel images:
• Local isometry image distance parameter space distance
• Linear tangent spacesare close approximationlocally
• Low dimensional articulation space
![Page 10: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/10.jpg)
Ex: Manifold Learning
LLEISOMAPLEHEDiff. Geo …
• K=1rotation
![Page 11: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/11.jpg)
Ex: Manifold Learning
• K=2rotation and scale
![Page 12: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/12.jpg)
Theory/Practice Disconnect Smoothness
• Practical image manifolds are not smooth!
• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]
Tangent approximations ?
![Page 13: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/13.jpg)
Theory/Practice Disconnect Smoothness
• Practical image manifolds are not smooth!
• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]
Tangent approximations ?
![Page 14: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/14.jpg)
Failure of Tangent Plane Approx.
• Ex: cross-fading when synthesizing / interpolating images that should lie on manifold
Input Image Input Image
Geodesic Linear path
![Page 15: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/15.jpg)
• Ex: translation manifold
all blue images are equidistant from the red image
• Local isometry
– satisfied only when sampling is dense
0 20 40 60 80 100
0
0.5
1
1.5
2
2.5
3
3.5
Translation q in [px]
Euc
lidea
n di
stan
ce
Theory/Practice Disconnect Isometry
![Page 16: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/16.jpg)
Theory/Practice DisconnectNuisance articulations
• Unsupervised data, invariably, has additional undesired articulations– Illumination– Background clutter, occlusions, …
• Image ensemble is no longer low-dimensional
![Page 17: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/17.jpg)
Image representations
• Conventional representation for an image– A vector of pixels– Inadequate!
pixel image
![Page 18: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/18.jpg)
Image representations
• Conventional representation for an image– A vector of pixels– Inadequate!
• Remainder of the talk– TWO novel image representations that alleviate the
theoretical/practical challenges in manifold learning on image ensembles
![Page 19: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/19.jpg)
Transport operators for image manifolds
![Page 20: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/20.jpg)
The concept ofTransport operators
))(()( xfIxIrefqq
Beyond point cloud model for image manifolds
refIfI qq
Example
![Page 21: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/21.jpg)
Example: Translation• 2D Translation manifold
• Set of all transport operators =• Beyond a point cloud model
– Action of the articulation is more accurate and meaningful
1I0I
2R
![Page 22: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/22.jpg)
Optical Flow
• Generalizing this idea: Pixel correspondances
• Idea:OF between two images is a natural and accurate transport operator
(Figures from Ce Liu’s optical flow page)
OFfrom I1 to I2
I1 and I2
![Page 23: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/23.jpg)
Optical Flow Transport
0q
1q 2q
IAM
2qI1q
I
Articulations
• Consider a reference imageand a K-dimensional articulation
• Collect optical flows fromto all images reachable by aK-dimensional articulation
![Page 24: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/24.jpg)
Optical Flow Transport
0q
1q 2q
IAM
2qI1q
I
Articulations
• Consider a reference imageand a K-dimensional articulation
• Collect optical flows fromto all images reachable by aK-dimensional articulation
![Page 25: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/25.jpg)
Optical Flow Transport
0q
1q 2q
IAM
OFM at 0qI
2qI1q
I0qI
Articulations
• Consider a reference imageand a K-dimensional articulation
• Collect optical flows fromto all images reachable by aK-dimensional articulation
• Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth)
for large class of articulations
![Page 26: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/26.jpg)
OFM is Smooth (Rotation)
00q 30q 60q 30q 60q
-80 -60 -40 -20 0 20 40 60 800
50
100
150
200
Articulation q in [ ]
Inte
nsity
-80 -60 -40 -20 0 20 40 60 80-20
-10
0
10
20
Articulation q in [ ]
Opt
ical
flow
vx i
n pi
xels
Articulation θ in [⁰]
Inte
nsity
I(θ)
Op. fl
ow v
(θ)
Pixel intensityat 3 points
Flow (nearly linear)
![Page 27: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/27.jpg)
Main results• Local model at each
• Each point on the OFM defines a transport operator– Each transport operator maps
to one of its neighbors
• For a large class of articulations, OFMs are smooth and locally isometric– Traditional manifold processing
techniques work on OFMs
0q
1q 2q
IAM
OFM at 0qI
2qI1q
I0qI
Articulations
0qI
0qI
![Page 28: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/28.jpg)
Linking it all together
0q
1q 2q
IAM
OFM at 0qI
2qI1q
I0qI
Articulations
0e
1e 2e
Nonlinear dim. reduction
The non-differentiability does not dissappear --- it is embedded in the mapping from OFM to the IAM.
However, this is a known map
))(()( such that },{ xfIxIIfI refref qqqq
![Page 29: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/29.jpg)
The Story So Far…
0q
1q 2q
IAM
OFM at 0qI
2qI1q
I0qI
Articulations
Tangent space at 0qI
Articulations
0q
1q 2q
2qI1q
I0qI IAM
![Page 30: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/30.jpg)
Input Image Input Image
Geodesic Linear pathIAM
OFM
![Page 31: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/31.jpg)
Data196 images of two bears moving linearlyand independently
IAM OFM
TaskFind low-dimensional embedding
OFM Manifold Learning
![Page 32: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/32.jpg)
IAM OFM
Data196 images of a cup moving on a plane
Task 1Find low-dimensional embedding
Task 2Parameter estimation for new images(tracing an “R”)
OFM ML + Parameter Estimation
![Page 33: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/33.jpg)
• Point on the manifold such that the sum of geodesic distances to every other point is minimized
• Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al]
Karcher Mean
10 images from an IAM
ground truth KM OFM KM linear KM
![Page 34: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/34.jpg)
Sparse keypoint-based image representation
![Page 35: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/35.jpg)
Image representations
• Conventional representation for an image– A vector of pixels
– Inadequate!
pixel image
![Page 36: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/36.jpg)
Image representations• Replace vector of pixels with an abstract
bag of features
– Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptors for each keypoint
![Page 37: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/37.jpg)
Image representations• Replace vector of pixels with an abstract
bag of features
– Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptors for each keypoint
– Feature descriptors are local; it is very easy to make them robust to nuisance imaging parameters
![Page 38: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/38.jpg)
Loss of Geometrical Info• Bag of features representations hide
potentially useful image geometry
• Goal: make salient image geometrical info more explicit for exploitation
Image space
Keypoint space
![Page 39: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/39.jpg)
Key idea• Keypoint space can be endowed with a rich
low-dimensional structure in many situations
• Mechanism: define kernels , between keypoint locations, keypoint descriptors
![Page 40: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/40.jpg)
Keypoint Kernel• Keypoint space can be endowed with a rich
low-dimensional structure in many situations
• Mechanism: define kernels , between keypoint locations, keypoint descriptors
• Joint keypoint kernel between two images
is given by
![Page 41: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/41.jpg)
Keypoint GeometryTheorem: Under the metric induced by the kernel
certain ensembles of articulating images formsmooth, isometric manifolds
• In contrast: conventional approach to image fusion via image articulation manifolds (IAMs) fraught with non-differentiability (due to sharp image edges)– not smooth– not isometric
![Page 42: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/42.jpg)
Application: Manifold Learning
2D Translation
![Page 43: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/43.jpg)
Application: Manifold Learning
2D Translation IAM KAM
![Page 44: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/44.jpg)
Manifold Learning in the Wild• Rice University’s Duncan Hall Lobby
– 158 images– 360° panorama using handheld camera– Varying brightness, clutter
![Page 45: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/45.jpg)
• Duncan Hall Lobby• Ground truth using state of the art
structure-from-motion software
Manifold Learning in the Wild
Ground truth IAM KAM
![Page 46: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/46.jpg)
Internet scale imagery• Notre-dame
cathedral – 738 images
– Collected from Flickr
– Large variations in illumination (night/day/saturations), clutter (people, decorations), camera parameters (focal length, fov, …)
– Non-uniform sampling of the space
![Page 47: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/47.jpg)
Organization• k-nearest neighbors
![Page 48: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/48.jpg)
Organization• “geodesics’
3D rotation
“Walk-closer”
“zoom-out”
![Page 49: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/49.jpg)
Summary• Need for novel image representations
– Transport operators Enables differentiability and meaningful transport
– Sparse features Robustness to outliers, nuisance articulations, etc. Learning in the wild: unsupervised imagery
• True power of manifold signal processing lies in fast algorithms that mainly use neighbor relationships– What are compelling applications where such methods can
achieve state of the art performance ?
![Page 50: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/50.jpg)
![Page 51: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/51.jpg)
Summary• IAMs a useful concise model for many image
processing problems involving image collections and multiple sensors/viewpoints
• But practical IAMs are non-differentiable– IAM-based algorithms have not lived up to their promise
• Optical flow manifolds (OFMs)– smooth even when IAM is not– OFM ~ nonlinear tangent space– support accurate image synthesis, learning, charting, …
• Barely discussed here: OF enables the safe extension of differential geometry concepts– Log/Exp maps, Karcher mean, parallel transport, …
![Page 52: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/52.jpg)
Summary: Rice U• Bag of features representations pervasive in
image information fusion applications (ex: SIFT)• Progress to date:
– Bags of features can contain rich geometrical structure – Keypoint distance very efficient to compute
(contrast with combinatorial complexity of feature correspondence algorithm)
– Keypoint distance significantly outperforms standard image Euclidean distance in learning and fusion applications
– Punch line: KAM preferred over IAM
• Current/future work:– More exhaustive experiments with occlusion and clutter– Studying geometrical structure of feature representations
of other kinds of non-image data (ex: documents)
![Page 53: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/53.jpg)
Open Questions
• Our treatment is specific to image manifolds under
brightness constancy
• What are the natural transport operators for other data manifolds?
dsp.rice.edu
![Page 54: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/54.jpg)
Related Work• Analytic transport operators
– transport operator has group structure [Xiao and Rao 07][Culpepper and Olshausen 09] [Miller and Younes 01] [Tuzel et al 08]
– non-linear analytics [Dollar et al 06]– spatio-temporal manifolds [Li and Chellappa 10]– shape manifolds [Klassen et al 04]
• Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups)
• In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations
![Page 55: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/55.jpg)
Limitations• Brightness constancy
– Optical flow is no longer meaningful
• Occlusion– Undefined pixel flow in theory, arbitrary flow estimates in
practice– Heuristics to deal with it
• Changing backgrounds etc.– Transport operator assumption too strict– Sparse correspondences ?
![Page 56: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/56.jpg)
![Page 57: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/57.jpg)
Occlusion• Detect occlusion using forward-backward flow
reasoning
• Remove occluded pixel computations
• Heuristic --- formal occlusion handling is hard
Occluded
![Page 58: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/58.jpg)
Open Questions• Theorem:
random measurements stably embed aK-dim manifoldwhp [B, Wakin, FOCM ’08]
• Q: Is there an analogousresult for OFMs?
![Page 59: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/59.jpg)
Tools for manifold processing
Geodesics, exponential maps, log-maps, Riemannian metrics, Karcher means, …
Smooth diff manifold
Algebraic manifolds Data manifolds
LLE, kNN graphs
Point cloud model
![Page 60: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/60.jpg)
![Page 61: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/61.jpg)
Concise Models• Efficient processing / compression requires
concise representation
• Sparsity of an individual image
pixels largewaveletcoefficients(blue = 0)
![Page 62: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/62.jpg)
History of Optical Flow
• Dark ages (<1985)– special cases solved– LBC an under-determined set of linear equations
• Horn and Schunk (1985) – Regularization term: smoothness prior on the flow
• Brox et al (2005)– shows that linearization of brightness constancy (BC) is
a bad assumption– develops optimization framework to handle BC directly
• Brox et al (2010), Black et al (2010), Liu et al (2010)– practical systems with reliable code
![Page 63: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/63.jpg)
ISOMAP embedding error for OFM and IAM
2D rotations
Reference image
Manifold Learning
![Page 64: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/64.jpg)
Manifold Learning
-6 -4 -2 0 2 4 6-5
-4
-3
-2
-1
0
1
2
3
4
5Two-dimensional Isomap embedding (with neighborhood graph).
Embedding of OFM
2D rotations
Reference image
![Page 65: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/65.jpg)
OFM Implementation details
Reference Image
![Page 66: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/66.jpg)
Pairwise distances and embedding
-5 0 5-4
-2
0
2
4
100 200 300 400
100
200
300
400
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
Res
idua
l var
ianc
e
Isomap dimensionality
![Page 67: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/67.jpg)
-5 0 5-4
-2
0
2
4100 200 300 400
100
200
300
400
0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
Res
idua
l var
ianc
e
Isomap dimensionality
Flow Embedding
![Page 68: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/68.jpg)
OFM Synthesis
![Page 69: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/69.jpg)
• Goal: build a generative model for an entire IAM/OFM based on a small number of base images
• eLCheapoTM algorithm:– choose a reference image randomly– find all images that can be generated from this image by OF– compute Karcher (geodesic) mean of these images– compute OF from Karcher mean image to other images– repeat on the remaining images until no images remain
• Exact representation when no occlusions
Manifold Charting
![Page 70: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/70.jpg)
Manifold Charting• Goal: build a generative model for an entire
IAM/OFM based on a small number of base images
• Ex: cube rotating about axis. All cube images can be representing using 4 reference images + OFMs
• Many applications– selection of target templates for classification– “next-view” selection for adaptive sensing applications
Greedy charting
Optimal charting
0q 90q 180q 180q 90q
IAM
![Page 71: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/71.jpg)
![Page 72: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/72.jpg)
• Features (including SIFT) ubiquitous in fusion and processing apps (15k+ cites for 2 SIFT papers)
SIFT Features
building 3D models
part-based object recognitionorganizing internet-scale databases
image stitching
Figures courtesy Rob Fergus (NYU), Phototourism website, Antonio Torralba (MIT), and Wei Lu
![Page 73: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/73.jpg)
Many Possible Kernels• Euclidean kernel
• Gaussian kernel
• Polynomial kernel
• Pyramid match kernel [Grauman et al. ’07]
• Many others
![Page 74: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/74.jpg)
Keypoint Kernel• Joint keypoint kernel between two images
is given by
• Using Euclidean/Gaussian (E/G) combination yields
![Page 75: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/75.jpg)
From Kernel to MetricTheorem: The E/G keypoint kernel is a Mercer kernel
– enables algorithms such as SVM
Theorem: The E/G keypoint kernel induces a metric on the space of images
– alternative to conventional L2 distance between images– keypoint metric robust to nuisance imaging parameters,
occlusion, clutter, etc.
![Page 76: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/76.jpg)
Keypoint Geometry
NOTE: Metric requires no permutation-based matching of keypoints(combinatorial complexity in general)
Theorem: The E/G keypoint kernel induces a metric on the space of images
![Page 77: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/77.jpg)
Keypoint GeometryTheorem: Under the metric induced by the kernel
certain ensembles of articulating images formsmooth, isometric manifolds
• Keypoint representation compact, efficient, and …
• Robust to illumination variations, non-stationary backgrounds, clutter, occlusions
![Page 78: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/78.jpg)
Manifold Learning in the Wild
Viewing angle – 179 images
IAM KAM
![Page 79: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/79.jpg)
Manifold Learning in the Wild• Rice University’s Brochstein Pavilion
– 400 outdoor images of a building– occlusions, movement in foreground, varying background
![Page 80: Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj](https://reader036.vdocument.in/reader036/viewer/2022081503/568163c9550346895dd50267/html5/thumbnails/80.jpg)
Manifold Learning in the Wild• Brochstein Pavilion
– 400 outdoor images of a building– occlusions, movement in foreground, background
IAM KAM