automatic image alignment (feature-based) 15-463: computational photography alexei efros, cmu, fall...
Post on 20-Dec-2015
219 views
TRANSCRIPT
![Page 1: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/1.jpg)
Automatic Image Alignment (feature-based)
15-463: Computational PhotographyAlexei Efros, CMU, Fall 2005
with a lot of slides stolen from Steve Seitz and Rick Szeliski
© Mike Nese
![Page 2: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/2.jpg)
Today’s lecture
• Feature detectors• scale invariant Harris corners
• Feature descriptors• patches, oriented patches
Reading for Project #4:
Multi-image Matching using Multi-scale image patches, CVPR 2005
![Page 3: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/3.jpg)
Invariant Local Features
Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters
Features Descriptors
![Page 4: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/4.jpg)
Advantages of local featuresLocality: features are local, so robust to occlusion and
clutter (no prior segmentation)
Distinctiveness: individual features can be matched to a large database of objects
Quantity: many features can be generated for even small objects
Efficiency: close to real-time performance
Extensibility: can easily be extended to wide range of differing feature types, with each adding robustness
![Page 5: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/5.jpg)
More motivation…
Feature points are used for:• Image alignment (homography, fundamental
matrix)• 3D reconstruction• Motion tracking• Object recognition• Indexing and database retrieval• Robot navigation• … other
![Page 6: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/6.jpg)
Harris corner detectorC.Harris, M.Stephens. “A Combined Corner and Edge
Detector”. 1988
![Page 7: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/7.jpg)
The Basic Idea
We should easily recognize the point by looking through a small window
Shifting a window in any direction should give a large change in intensity
![Page 8: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/8.jpg)
Harris Detector: Basic Idea
“flat” region:no change in all directions
“edge”:no change along the edge direction
“corner”:significant change in all directions
![Page 9: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/9.jpg)
Harris Detector: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
Change of intensity for the shift [u,v]:
IntensityShifted intensity
Window function
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
![Page 10: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/10.jpg)
Harris Detector: Mathematics
( , ) ,u
E u v u v Mv
For small shifts [u,v] we have a bilinear approximation:
2
2,
( , ) x x y
x y x y y
I I IM w x y
I I I
where M is a 22 matrix computed from image derivatives:
![Page 11: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/11.jpg)
Harris Detector: Mathematics
1
2
“Corner”1 and 2 are large,
1 ~ 2;
E increases in all directions
1 and 2 are small;
E is almost constant in all directions
“Edge” 1 >> 2
“Edge” 2 >> 1
“Flat” region
Classification of image points using eigenvalues of M:
![Page 12: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/12.jpg)
Harris Detector: Mathematics
Measure of corner response:
1 2
1 2
det
trace
M
M
M
MR
Trace
det
![Page 13: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/13.jpg)
Harris Detector
The Algorithm:• Find points with large corner response function R
(R > threshold)• Take the points of local maxima of R
![Page 14: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/14.jpg)
Harris Detector: Workflow
![Page 15: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/15.jpg)
Harris Detector: Workflow
Compute corner response R
![Page 16: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/16.jpg)
Harris Detector: Workflow
Find points with large corner response: R>threshold
![Page 17: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/17.jpg)
Harris Detector: Workflow
Take only the points of local maxima of R
![Page 18: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/18.jpg)
Harris Detector: Workflow
![Page 19: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/19.jpg)
Harris Detector: Some PropertiesRotation invariance
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response R is invariant to image rotation
![Page 20: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/20.jpg)
Harris Detector: Some PropertiesPartial invariance to affine intensity change
Only derivatives are used => invariance to intensity shift I I + b
Intensity scale: I a I
R
x (image coordinate)
threshold
R
x (image coordinate)
![Page 21: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/21.jpg)
Harris Detector: Some Properties
But: non-invariant to image scale!
All points will be classified as edges
Corner !
![Page 22: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/22.jpg)
Scale Invariant Detection
Consider regions (e.g. circles) of different sizes around a point
Regions of corresponding sizes will look the same in both images
![Page 23: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/23.jpg)
Scale Invariant Detection
The problem: how do we choose corresponding circles independently in each image?
Choose the scale of the “best” corner
![Page 24: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/24.jpg)
Feature selectionDistribute points evenly over the image
![Page 25: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/25.jpg)
Adaptive Non-maximal Suppression
Desired: Fixed # of features per image• Want evenly distributed spatially…
• Search over non-maximal suppression radius[Brown, Szeliski, Winder, CVPR’05]
![Page 26: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/26.jpg)
Feature descriptorsWe know how to detect pointsNext question: How to match them?
?
Point descriptor should be:1. Invariant 2. Distinctive
![Page 27: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/27.jpg)
Descriptors Invariant to Rotation
Find local orientation
Dominant direction of gradient
• Extract image patches relative to this orientation
![Page 28: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/28.jpg)
Multi-Scale Oriented PatchesInterest points
• Multi-scale Harris corners• Orientation from blurred gradient• Geometrically invariant to rotation
Descriptor vector• Bias/gain normalized sampling of local patch (8x8)• Photometrically invariant to affine changes in intensity
[Brown, Szeliski, Winder, CVPR’2005]
![Page 29: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/29.jpg)
Descriptor VectorOrientation = blurred gradient
Rotation Invariant Frame• Scale-space position (x, y, s) + orientation ()
![Page 30: Automatic Image Alignment (feature-based) 15-463: Computational Photography Alexei Efros, CMU, Fall 2005 with a lot of slides stolen from Steve Seitz and](https://reader036.vdocument.in/reader036/viewer/2022062516/56649d4f5503460f94a2ea48/html5/thumbnails/30.jpg)
Detections at multiple scales