sift-based image alignmentfeature space outlier rejection •how can we tell which putative matches...
TRANSCRIPT
![Page 1: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/1.jpg)
SIFT-based Image Alignment
Jianping Fan
Dept of Computer Science
UNC-Charlotte
Course Website: http://webpages.uncc.edu/jfan/itcs5152.html
![Page 2: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/2.jpg)
Project 2: SIFT-based Image Alignment
![Page 3: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/3.jpg)
Correspondence and Alignment
• Correspondence: matching points, patches, edges, or regions across images
≈
![Page 4: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/4.jpg)
DoG for
identifying scale-
invariant local
extrema
Extrema points
Keypoints after removing
low contrast & edge points
Keypoints & SIFT Descriptors
![Page 5: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/5.jpg)
Keypoint & SIFT Descriptor• 16x16 Gradient window is taken. Partitioned into 4x4
subwindows.
• Histogram of 4x4 samples in 8 directions
• Gaussian weighting around center( is 0.5 times that of the scale of a keypoint)
• 4x4x8 = 128 dimensional feature vector
Keypoint Detection
& Orientation Determination
& Neighborhood Pattern
![Page 6: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/6.jpg)
Image alignment
www.cs.unc.edu/~lazebnik/spring10/lec10_alignment
![Page 7: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/7.jpg)
Image Alignment Algorithm
Given images A and B
1. Compute image features for A and B
2. Match features between A and B
3. Compute homography between A and B using least squares on set of matches
What could go wrong?
![Page 8: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/8.jpg)
A look into the past
http://blog.flickr.net/en/2010/01/27/a-look-into-the-past/
![Page 9: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/9.jpg)
A look into the past
• Leningrad during the blockade
http://komen-dant.livejournal.com/345684.html
![Page 10: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/10.jpg)
Bing streetside images
http://www.bing.com/community/blogs/maps/archive/2010/01/12/new-bing-
maps-application-streetside-photos.aspx
![Page 11: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/11.jpg)
Image alignment: Applications
Panorama stitching
Recognition
of object
instances
![Page 12: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/12.jpg)
Image alignment: Challenges
Small degree of overlap
Occlusion,
clutter
Intensity changes
![Page 13: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/13.jpg)
Image alignment
• Two broad approaches:– Direct (pixel-based) alignment
• Search for alignment where most pixels agree
– Feature-based alignment• Search for alignment where extracted features
agree
• Can be verified using pixel-based alignment
![Page 14: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/14.jpg)
Image Alignment as Fitting
![Page 15: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/15.jpg)
Alignment as fitting
• Previous lectures: fitting a model to features in one image
i
i Mx ),(residual
Find model M that minimizes
M
xi
![Page 16: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/16.jpg)
Alignment as fitting• Fitting a model to features in one image
• Alignment: fitting a model to a transformation between pairs of features (matches) in two images
i
i Mx ),(residual
i
ii xxT )),((residual
Find model M that minimizes
Find transformation T
that minimizes
M
xi
T
xixi
'
![Page 17: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/17.jpg)
2D transformation models
• Similarity(translation, scale, rotation)
• Affine
• Projective(homography)
![Page 18: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/18.jpg)
Parametric (global) warping
Transformation T is a coordinate-changing machine:p’ = T(p)
What does it mean that T is global?– Is the same for any point p– can be described by just a few numbers (parameters)
For linear transformations, we can represent T as a matrixp’ = Tp
T
p = (x,y) p’ = (x’,y’)
y
x
y
xT
'
'
![Page 19: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/19.jpg)
Common transformations
translation rotation aspect
affine perspective
original
Transformed
Slide credit (next few slides):
A. Efros and/or S. Seitz
![Page 20: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/20.jpg)
Transformations
translation
Original Image Transformed Image
𝑥′ = 𝑥 + 𝑣
𝑦′ = 𝑦 + 𝑢
![Page 21: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/21.jpg)
Transformations
Rotation
Original Image Transformed Image
y
x
y
x
cossin
sincos
'
'
![Page 22: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/22.jpg)
Transformations
Aspect
Original Image Transformed Image𝑥′ = 𝑥/𝑟
𝑦′ = 𝑦/𝑡
y
x
b
a
y
x
0
0
'
'
![Page 23: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/23.jpg)
Transformations
Affine
Original ImageTransformed Image
1
y
x
fed
cba
y
x
![Page 24: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/24.jpg)
Transformations
Original ImageTransformed Image
Perspective
w
yx
ihg
fedcba
w
yx
'
''
![Page 25: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/25.jpg)
Scaling• Scaling a coordinate means multiplying each of its components by a
scalar
• Uniform scaling means this scalar is the same for all components:
2
![Page 26: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/26.jpg)
• Non-uniform scaling: different scalars per component:
Scaling
X 2,
Y 0.5
![Page 27: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/27.jpg)
Scaling
• Scaling operation:
• Or, in matrix form:
byy
axx
'
'
y
x
b
a
y
x
0
0
'
'
scaling matrix S
![Page 28: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/28.jpg)
2-D Rotation
(x, y)
(x’, y’)
x’ = x cos() - y sin()
y’ = x sin() + y cos()
![Page 29: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/29.jpg)
2-D RotationThis is easy to capture in matrix form:
Even though sin() and cos() are nonlinear functions of ,
– x’ is a linear combination of x and y
– y’ is a linear combination of x and y
What is the inverse transformation?
– Rotation by –
– For rotation matrices
y
x
y
x
cossin
sincos
'
'
TRR 1
R
![Page 30: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/30.jpg)
Basic 2D transformations
TranslateRotate
ShearScale
y
x
y
x
y
x
1
1
'
'
y
x
y
x
cossin
sincos
'
'
y
x
s
s
y
x
y
x
0
0
'
'
110
01y
x
t
t
y
x
y
x
1
y
x
fed
cba
y
x
Affine
Affine is any combination of
translation, scale, rotation,
shear
![Page 31: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/31.jpg)
Affine Transformations
Affine transformations are combinations of
• Linear transformations, and
• Translations
Properties of affine transformations:
• Lines map to lines
• Parallel lines remain parallel
• Ratios are preserved
• Closed under composition
1
y
x
fed
cba
y
x
11001
'
'
y
x
fed
cba
y
x
or
![Page 32: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/32.jpg)
Projective Transformations
w
yx
ihg
fedcba
w
yx
'
''Projective transformations are combos of
• Affine transformations, and
• Projective warps
Properties of projective transformations:
• Lines map to lines
• Parallel lines do not necessarily remain parallel
• Ratios are not preserved
• Closed under composition
• Models change of basis
• Projective matrix is defined up to a scale (8 DOF)
![Page 33: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/33.jpg)
2D image transformations (reference table)
Szeliski 2.1
![Page 34: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/34.jpg)
Image matching under transformation
Transformation T is a coordinate-changing machine:p’ = T(p)
What does it mean that T is global?– Is the same for any point p– can be described by just a few numbers (parameters)
For linear transformations, we can represent T as a matrixp’ = Tp
T
p = (x,y) p’ = (x’,y’)
y
x
y
xT
'
'
![Page 35: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/35.jpg)
affine transformations
• Simple fitting procedure (linear least squares)
• Approximates viewpoint changes for roughly planar objects and roughly orthographic cameras
• Can be used to initialize fitting for more complex models
![Page 36: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/36.jpg)
Fitting an affine transformation
• Assume we know the correspondences, how do we get the transformation?
),( ii yx ),( ii yx
2
1
43
21
t
t
y
x
mm
mm
y
x
i
i
i
i
i
i
ii
ii
y
x
t
t
m
m
m
m
yx
yx
2
1
4
3
2
1
1000
0100
![Page 37: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/37.jpg)
Fitting an affine transformation
• Linear system with six unknowns
• Each match gives us two linearly independent equations: need at least three to solve for the transformation parameters
i
i
ii
ii
y
x
t
t
m
m
m
m
yx
yx
2
1
4
3
2
1
1000
0100
![Page 38: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/38.jpg)
Alignment as fitting• Transformation between pairs of features
(matches) in two images
i
ii xxT )),((residual
Find transformation T
that minimizes
T
xi
xi'
1
y
x
fed
cba
y
x
T
![Page 39: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/39.jpg)
Hough Transformation for Alignment
![Page 40: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/40.jpg)
Hough transform
• Recall: Generalized Hough transform
B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with
an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004
model
visual codeword with
displacement vectors
test image
![Page 41: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/41.jpg)
Hough transform• Suppose our features are adapted to scale and rotation
– Then a single feature match provides an alignment hypothesis (translation, scale, orientation)
David G. Lowe. "Distinctive image features from scale-invariant keypoints.”
IJCV 60 (2), pp. 91-110, 2004.
model
![Page 42: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/42.jpg)
Hough transform• Suppose our features are adapted to scale and rotation
– Then a single feature match provides an alignment hypothesis (translation, scale, orientation)
– Of course, a hypothesis obtained from a single match is unreliable
– Solution: let each match vote for its hypothesis in a Hough space with very coarse bins
David G. Lowe. "Distinctive image features from scale-invariant keypoints.”
IJCV 60 (2), pp. 91-110, 2004.
model
![Page 43: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/43.jpg)
Hough transform details (D. Lowe’s system)
• Modeling phase: For each model feature, record 2D location, scale, and orientation of model (relative to normalized feature frame)
• Test phase: Let each match between a test and a model feature vote in a 4D Hough space– Use broad bin sizes of 30 degrees for orientation, a factor
of 2 for scale, and 0.25 times image size for location– Vote for two closest bins in each dimension
• Find all bins with at least three votes and perform geometric verification – Estimate least squares affine transformation – Use stricter thresholds on transformation residual– Search for additional features that agree with the
alignment
David G. Lowe. "Distinctive image features from scale-invariant keypoints.”
IJCV 60 (2), pp. 91-110, 2004.
![Page 44: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/44.jpg)
Features for Image Alignment
![Page 45: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/45.jpg)
Feature-based alignment outline
![Page 46: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/46.jpg)
Feature-based alignment outline
• Extract features
![Page 47: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/47.jpg)
Feature-based alignment outline
• Extract features
• Compute putative matches
![Page 48: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/48.jpg)
Feature-based alignment outline
• Extract features
• Compute putative matches
• Loop:
– Hypothesize transformation T
![Page 49: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/49.jpg)
Feature-based alignment outline
• Extract features
• Compute putative matches
• Loop:– Hypothesize transformation T
– Verify transformation (search for other matches consistent with T)
![Page 50: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/50.jpg)
Feature-based alignment outline
• Extract features
• Compute putative matches
• Loop:– Hypothesize transformation T
– Verify transformation (search for other matches consistent with T)
![Page 51: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/51.jpg)
Generating putative correspondences
?
![Page 52: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/52.jpg)
Generating putative correspondences
• Need to compare feature descriptors of local patches surrounding interest points
( ) ( )=?
feature
descriptorfeature
descriptor
?
![Page 53: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/53.jpg)
Feature descriptors• Assuming the patches are already normalized
(i.e., the local effect of the geometric transformation is factored out), how do we compute their similarity?
• Want invariance to intensity changes, noise, perceptually insignificant changes of the pixel pattern
![Page 54: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/54.jpg)
• Simplest descriptor: vector of raw intensity values• How to compare two such vectors?
– Sum of squared differences (SSD)
• Not invariant to intensity change
– Normalized correlation
• Invariant to affine intensity change
Feature descriptors
i
ii vuvu2
),SSD(
j
j
j
j
i ii
vvuu
vvuuvu
22 )()(
))((),(
![Page 55: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/55.jpg)
Feature descriptors• Disadvantage of patches as descriptors:
– Small shifts can affect matching score a lot
• Solution: histograms
0 2 p
![Page 56: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/56.jpg)
• Descriptor computation:
– Divide patch into 4x4 sub-patches
– Compute histogram of gradient orientations (8 reference angles) inside each sub-patch
– Resulting descriptor: 4x4x8 = 128 dimensions
Feature descriptors: SIFT
David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60
(2), pp. 91-110, 2004.
![Page 57: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/57.jpg)
• Descriptor computation:– Divide patch into 4x4 sub-patches
– Compute histogram of gradient orientations (8 reference angles) inside each sub-patch
– Resulting descriptor: 4x4x8 = 128 dimensions
• Advantage over raw vectors of pixel values– Gradients less sensitive to illumination change
– Pooling of gradients over the sub-patches achieves robustness to small shifts, but still preserves some spatial information
Feature descriptors: SIFT
David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60
(2), pp. 91-110, 2004.
![Page 58: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/58.jpg)
Feature matching
?
• Generating putative matches: for each patch in one image, find a short list of patches in the other image that could match it based solely on appearance
![Page 59: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/59.jpg)
Feature space outlier rejection
• How can we tell which putative matches are more reliable?
• Heuristic: compare distance of nearest neighbor to that of second nearest neighbor– Ratio of closest distance to second-closest distance will
be high for features that are not distinctive
David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60
(2), pp. 91-110, 2004.
Threshold of 0.8
provides good
separation
![Page 60: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/60.jpg)
Reading
David G. Lowe. "Distinctive image features from scale-
invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004.
![Page 61: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/61.jpg)
RANSAC Technique
![Page 62: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/62.jpg)
What’s RANSAC ?
• RANSAC is an abbreviation for "RANdomSAmple Consensus".
• It is an iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliers.
• Non-deterministic algorithm.
63http://en.wikipedia.org/wiki/RANSAC
![Page 63: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/63.jpg)
Why RANSAC ?
• RANSAC can estimate a model which ignored outliers.
• Example:
– To fit a line
• Least Squares method: – Optimally fitted to all points including outliers.
• RANSAC:– Only computed from the inliers.
64http://en.wikipedia.org/wiki/RANSAC
Inliers vs. Outliers
![Page 64: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/64.jpg)
RANSAC
General version:
1. Randomly choose s samples
• Typically s = minimum sample size that lets you fit a model
2. Fit a model (e.g., line) to those samples
3. Count the number of inliers that approximately fit the model
4. Repeat N times
5. Choose the model that has the largest set of inliers
![Page 65: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/65.jpg)
How big is s?
• For alignment, depends on the motion model– Here, each sample is a correspondence (pair of
matching points)
![Page 66: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/66.jpg)
RANSAC pros and cons
• Pros
– Simple and general
– Applicable to many different problems
– Often works well in practice
• Cons
– Parameters to tune
– Sometimes too many iterations are required
– Can fail for extremely low inlier ratios
– We can often do better than brute-force sampling
![Page 67: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/67.jpg)
Illustration of RANSAC
68cmp.felk.cvut.cz/~matas/papers/presentations/viva.ppt
![Page 68: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/68.jpg)
Illustration of RANSAC
69
• Select sample of m
points at random
cmp.felk.cvut.cz/~matas/papers/presentations/viva.ppt
![Page 69: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/69.jpg)
Illustration of RANSAC
70
• Select sample of m points
at random
• Calculate model
parameters that fit the
data in the sample
cmp.felk.cvut.cz/~matas/papers/presentations/viva.ppt
![Page 70: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/70.jpg)
Illustration of RANSAC
71cmp.felk.cvut.cz/~matas/papers/presentations/viva.ppt
![Page 71: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/71.jpg)
Illustration of RANSAC
72
• Select sample of m points
at random
• Calculate model
parameters that fit the data
in the sample
• Calculate error function
for each data point
• Select data that support
current hypothesis
cmp.felk.cvut.cz/~matas/papers/presentations/viva.ppt
![Page 72: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/72.jpg)
Illustration of RANSAC
73
• Select sample of m points
at random
• Calculate model
parameters that fit the data
in the sample
• Calculate error function
for each data point
• Select data that support
current hypothesis
• Repeat sampling
cmp.felk.cvut.cz/~matas/papers/presentations/viva.ppt
![Page 73: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/73.jpg)
Illustration of RANSAC
74
• Select sample of m points
at random
• Calculate model
parameters that fit the data
in the sample
• Calculate error function
for each data point
• Select data that support
current hypothesis
• Repeat sampling
cmp.felk.cvut.cz/~matas/papers/presentations/viva.ppt
![Page 74: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/74.jpg)
Illustration of RANSAC
75
RANSAC time complexity
k … number of samples
drawn
N … number of data points
tM … time to compute a single
model
mS … average number of
models per sample
ALL-INLIER SAMPLE
cmp.felk.cvut.cz/~matas/papers/presentations/viva.ppt
![Page 75: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/75.jpg)
RANSAC Algorithm• Input:
– data: a set of observations– model: a model that can be fitted to data– n: the minimum number of data required to fit the model– k: the maximum number of iterations allowed in the
algorithm – t: a threshold value for determining when a datum fits a
model– d: the number of close data values required to assert that
a model fits well to data• Output:
– best_model : model parameters which best fit the data (or nil if no good model is found)
– best_consensus_set : data point from which this model has been estimated
– best_error : the error of this model relative to the data76
http://en.wikipedia.org/wiki/RANSAC
![Page 76: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/76.jpg)
RANSAC Algorithm
77http://en.wikipedia.org/wiki/RANSAC
![Page 77: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/77.jpg)
Parameters
78
knwp 11
)1log(
1lognw
pk
k: Iteration times.
n: Selected points in one iteration.
p: Probability in k iteration selects
only inliers.
w: Probability of a point which is
a inlier.
In general, the p is unknown. If we fixed p,
the k increased when n increased.
http://en.wikipedia.org/wiki/RANSAC
![Page 78: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/78.jpg)
RANSAC for Image Alignment
![Page 79: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/79.jpg)
RANSACRANSAC loop:
1. Randomly select a seed group of matches
2. Compute transformation from seed group
3. Find inliers to this transformation
4. If the number of inliers is sufficiently large, re-compute least-squares estimate of transformation on all of the inliers
• Keep the transformation with the largest number of inliers
![Page 80: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/80.jpg)
RANSAC example: Translation
Putative matches
![Page 81: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/81.jpg)
RANSAC example: Translation
Select one match, count inliers
![Page 82: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/82.jpg)
RANSAC example: Translation
Select one match, count inliers
![Page 83: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/83.jpg)
RANSAC example: Translation
Select translation with the most inliers
![Page 84: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/84.jpg)
Problem with RANSAC
• In many practical situations, the percentage of outliers (incorrect putative matches) is often very high (90% or above)
• Alternative strategy: Hough transform
![Page 85: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/85.jpg)
RANSAC
Algorithm:
1. Sample (randomly) the number of points required to fit the model
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model
Repeat 1-3 until the best model is found with high confidence
Fischler & Bolles in ‘81.
(RANdom SAmple Consensus) :
![Page 86: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/86.jpg)
RANSAC
Algorithm:
1. Sample (randomly) the number of points required to fit the model (#=2)
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model
Repeat 1-3 until the best model is found with high confidence
Illustration by Savarese
Line fitting example
![Page 87: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/87.jpg)
RANSAC
Algorithm:
1. Sample (randomly) the number of points required to fit the model (#=2)
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model
Repeat 1-3 until the best model is found with high confidence
Line fitting example
![Page 88: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/88.jpg)
RANSAC
6IN
Algorithm:
1. Sample (randomly) the number of points required to fit the model (#=2)
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model
Repeat 1-3 until the best model is found with high confidence
Line fitting example
![Page 89: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/89.jpg)
RANSAC
14INAlgorithm:
1. Sample (randomly) the number of points required to fit the model (#=2)
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model
Repeat 1-3 until the best model is found with high confidence
![Page 90: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/90.jpg)
How to choose parameters?• Number of samples N
– Choose N so that, with probability p, at least one random sample is free from outliers (e.g. p=0.99) (outlier ratio: e )
• Number of sampled points s– Minimum number needed to fit the model
• Distance threshold – Choose so that a good point with noise is likely (e.g., prob=0.95) within threshold
– Zero-mean Gaussian noise with std. dev. σ: t2=3.84σ2
se11log/p1logN
proportion of outliers e
s 5% 10% 20% 25% 30% 40% 50%
2 2 3 5 6 7 11 17
3 3 4 7 9 11 19 35
4 3 5 9 13 17 34 72
5 4 6 12 17 26 57 146
6 4 7 16 24 37 97 293
7 4 8 20 33 54 163 588
8 5 9 26 44 78 272 1177
modified from M. PollefeysFor p = 0.99
![Page 91: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/91.jpg)
RANSAC conclusions
Good• Robust to outliers• Applicable for larger number of model parameters than
Hough transform• Optimization parameters are easier to choose than Hough
transform
Bad• Computational time grows quickly with fraction of outliers
and number of parameters • Not good for getting multiple fits
Common applications• Computing a homography (e.g., image stitching)• Estimating fundamental matrix (relating two views)
![Page 92: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/92.jpg)
Alignment
• Alignment: find parameters of model that maps one set of points to another
• Typically want to solve for a global transformation that accounts for *most* true correspondences
• Difficulties
– Noise (typically 1-3 pixels)
– Outliers (often 50%)
– Many-to-one matches or multiple objects
![Page 93: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/93.jpg)
Example: solving for translation
A1
A2 A3B1
B2 B3
Given matched points in {A} and {B}, estimate the translation of the object
y
x
A
i
A
i
B
i
B
i
t
t
y
x
y
x
![Page 94: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/94.jpg)
Example: solving for translation
A1
A2 A3B1
B2 B3
Least squares solution
y
x
A
i
A
i
B
i
B
i
t
t
y
x
y
x
(tx, ty)
1. Write down objective function
2. Derived solution
a) Compute derivative
b) Compute solution
3. Computational solution
a) Write in form Ax=b
b) Solve using pseudo-inverse or
eigenvalue decomposition
A
n
B
n
A
n
B
n
AB
AB
y
x
yy
xx
yy
xx
t
t
11
11
10
01
10
01
![Page 95: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/95.jpg)
Example: solving for translation
A1
A2 A3B1
B2 B3
RANSAC solution
y
x
A
i
A
i
B
i
B
i
t
t
y
x
y
x
(tx, ty)
1. Sample a set of matching points (1 pair)
2. Solve for transformation parameters
3. Score parameters with number of inliers
4. Repeat steps 1-3 N times
Problem: outliers
A4
A5
B5
B4
![Page 96: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/96.jpg)
Example: solving for translation
A1
A2 A3B1
B2 B3
Hough transform solution
y
x
A
i
A
i
B
i
B
i
t
t
y
x
y
x
(tx, ty)
1. Initialize a grid of parameter values
2. Each matched pair casts a vote for
consistent values
3. Find the parameters with the most votes
4. Solve using least squares with inliers
A4
A5 A6
B4
B5 B6
Problem: outliers, multiple objects, and/or many-to-one matches
![Page 97: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/97.jpg)
Example: solving for translation
(tx, ty)
Problem: no initial guesses for correspondence
y
x
A
i
A
i
B
i
B
i
t
t
y
x
y
x
![Page 98: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/98.jpg)
ICP for Image Alignment
![Page 99: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/99.jpg)
Iterative Closest Points (ICP) Algorithm
Goal: estimate transform between two dense sets of points
1. Initialize transformation (e.g., compute difference in means and scale)
2. Assign each point in {Set 1} to its nearest neighbor in {Set 2}
3. Estimate transformation parameters – e.g., least squares or robust least squares
4. Transform the points in {Set 1} using estimated parameters
5. Repeat steps 2-4 until change is very small
![Page 100: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/100.jpg)
Example: aligning boundaries1. Extract edge pixels 𝑝1. . 𝑝𝑛 and 𝑞1. . 𝑞𝑚2. Compute initial transformation (e.g., compute translation and scaling
by center of mass, variance within each image)
3. Get nearest neighbors: for each point 𝑝𝑖 find corresponding match(i) = argmin
𝑗𝑑𝑖𝑠𝑡(𝑝𝑖, 𝑞𝑗)
4. Compute transformation T based on matches
5. Warp points p according to T
6. Repeat 3-5 until convergence
pq
![Page 101: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/101.jpg)
Example: solving for translation
(tx, ty)
Problem: no initial guesses for correspondence
y
x
A
i
A
i
B
i
B
i
t
t
y
x
y
xICP solution1. Find nearest neighbors for each point
2. Compute transform using matches
3. Move points using transform
4. Repeat steps 1-3 until convergence
![Page 102: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/102.jpg)
Applications of Feature Matching
![Page 103: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/103.jpg)
Scalability: Alignment to large databases• What if we need to align a test image with
thousands or millions of images in a model database?
– Efficient putative match generation
• Fast nearest neighbor search, inverted indexes
Model
database
?
Test image
![Page 104: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/104.jpg)
Scalability of SIFT Matching
![Page 105: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/105.jpg)
Scalability: Alignment to large databases• What if we need to align a test image with
thousands or millions of images in a model database?
– Efficient putative match generation
• Fast nearest neighbor search, inverted indexes
D. Nistér and H. Stewénius, Scalable
Recognition with a Vocabulary Tree,
CVPR 2006
Test image
Database
Vocabulary
tree with
inverted
index
![Page 106: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/106.jpg)
Descriptor spaceSlide credit: D. Nister
![Page 107: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/107.jpg)
Hierarchical partitioning of descriptor space (vocabulary tree)
Slide credit: D. Nister
![Page 108: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/108.jpg)
Slide credit: D. NisterVocabulary tree/inverted index
![Page 109: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/109.jpg)
Populating the vocabulary tree/inverted indexSlide credit: D. Nister
Model images
![Page 110: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/110.jpg)
Populating the vocabulary tree/inverted indexSlide credit: D. Nister
Model images
![Page 111: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/111.jpg)
Populating the vocabulary tree/inverted indexSlide credit: D. Nister
Model images
![Page 112: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/112.jpg)
Populating the vocabulary tree/inverted indexSlide credit: D. Nister
Model images
![Page 113: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/113.jpg)
Looking up a test imageSlide credit: D. Nister
Test imageModel images
![Page 114: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/114.jpg)
Indexing with geometric invariants• A match between invariant descriptors can yield a transformation
hypothesis
model
index
![Page 115: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/115.jpg)
Indexing with geometric invariants• A match between invariant descriptors can yield a transformation
hypothesis
model
test image
index
![Page 116: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/116.jpg)
Indexing with geometric invariants• When we don’t have feature descriptors, we can take n-tuples of
neighboring features and compute invariant features from their geometric configurations
• Application: searching the sky: http://www.astrometry.net/
A
B
C
D
![Page 117: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/117.jpg)
Projective (Homography) Transformation
![Page 118: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/118.jpg)
Beyond affine transformations
• Homography: plane projective transformation (transformation taking a quad to another arbitrary quad)
![Page 119: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/119.jpg)
Homography• The transformation between two views of a
planar surface
• The transformation between images from two cameras that share the same center
![Page 120: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/120.jpg)
Application: Panorama stitching
![Page 121: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/121.jpg)
Fitting a homography
• Recall: homogenenous coordinates
Converting to homogenenous
image coordinates
Converting from homogenenous
image coordinates
![Page 122: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/122.jpg)
Fitting a homography
• Recall: homogeneous coordinates
• Equation for homography:
Converting to homogeneous
image coordinates
Converting from homogeneous
image coordinates
11 333231
232221
131211
y
x
hhh
hhh
hhh
y
x
![Page 123: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/123.jpg)
Fitting a homography
• Equation for homography:
ii xHx
11 333231
232221
131211
i
i
i
i
y
x
hhh
hhh
hhh
y
x
0 ii xHx
i
T
ii
T
i
i
T
ii
T
i
T
i
T
i
i
T
i
T
i
T
i
i
yx
x
y
y
x
xhxh
xhxh
xhxh
xh
xh
xh
3
2
1
12
31
23
1
0
0
0
0
3
2
1
h
h
h
xx
xx
xx
TT
ii
T
ii
T
ii
TT
i
T
ii
T
i
T
xy
x
y3 equations,
only 2 linearly
independent
![Page 124: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/124.jpg)
Direct linear transform
• H has 8 degrees of freedom (9 parameters, but scale is arbitrary)
• One match gives us two linearly independent equations
• Four matches needed for a minimal solution (null space of 8x9 matrix)
• More than four: homogeneous least squares
0
0
0
0
0
3
2
1111
111
h
h
h
xx
xx
xx
xx
T
nn
TT
n
T
nn
T
n
T
TTT
TTT
x
y
x
y
0hA
![Page 125: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/125.jpg)
RANSAC for Estimating Homography
RANSAC loop:
1. Select four feature pairs (at random)
2. Compute homography H (exact)
3. Compute inliers where SSD(pi’, H pi) < ε
4. Keep largest set of inliers
5. Re-compute least-squares H estimate on all of the inliers
![Page 126: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/126.jpg)
RANSAC
![Page 127: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/127.jpg)
Why “Recognising Panoramas”?
• 1D Rotations ()
– Ordering matching images
Computational Photography, Alexei Efros, CMU, Fall 2005
![Page 128: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/128.jpg)
Why “Recognising Panoramas”?
• 1D Rotations ()
– Ordering matching images
![Page 129: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/129.jpg)
Why “Recognising Panoramas”?
• 2D Rotations (, f)
– Ordering matching images
• 1D Rotations ()
– Ordering matching images
![Page 130: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/130.jpg)
Why “Recognising Panoramas”?
• 1D Rotations ()
– Ordering matching images
• 2D Rotations (, f)
– Ordering matching images
![Page 131: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/131.jpg)
Why “Recognising Panoramas”?
• 1D Rotations ()
– Ordering matching images
• 2D Rotations (, f)
– Ordering matching images
![Page 132: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/132.jpg)
Why “Recognising Panoramas”?
![Page 133: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/133.jpg)
Overview of Image Alignment
• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
Computational Photography, Alexei Efros, CMU, Fall 2005
![Page 134: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/134.jpg)
RANSAC for Homography
![Page 135: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/135.jpg)
RANSAC for Homography
![Page 136: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/136.jpg)
RANSAC for Homography
![Page 137: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/137.jpg)
Probabilistic model for verification
![Page 138: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/138.jpg)
Finding the panoramas
![Page 139: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/139.jpg)
Finding the panoramas
![Page 140: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/140.jpg)
Finding the panoramas
![Page 141: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/141.jpg)
Finding the panoramas
![Page 142: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/142.jpg)
• Parameterise each camera by rotation and focal length
• This gives pairwise homographies
Homography for Rotation
![Page 143: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/143.jpg)
Bundle Adjustment
• New images initialised with rotation, focal length of best matching image
Computational Photography, Alexei Efros, CMU, Fall 2005
![Page 144: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/144.jpg)
Bundle Adjustment
• New images initialised with rotation, focal length of best matching image
Computational Photography, Alexei Efros, CMU, Fall 2005
![Page 145: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/145.jpg)
Multi-band Blending
• Burt & Adelson 1983
– Blend frequency bands over range
![Page 146: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/146.jpg)
Results
![Page 147: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/147.jpg)
• In order to figure this out, we need to learn what a camerais
Can we use homographies to create a 360 panorama?
![Page 148: SIFT-based Image AlignmentFeature space outlier rejection •How can we tell which putative matches are more reliable? •Heuristic: compare distance of nearest neighbor to that of](https://reader035.vdocument.in/reader035/viewer/2022071423/611d1487eccc9d6b4f72173f/html5/thumbnails/148.jpg)
360 panorama