project 3 questions?
DESCRIPTION
Project 3 questions?. 10/21/11. Interest Points and Instance Recognition. Computer Vision CS 143, Brown James Hays. Many slides from Kristen Grauman and Derek Hoiem. Last time. Detecting corner-like points in an image. Today. Local invariant features Detection of interest points - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/1.jpg)
![Page 2: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/2.jpg)
Project 3 questions?
![Page 3: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/3.jpg)
Interest Points and Instance Recognition
Computer VisionCS 143, Brown
James Hays
10/21/11
Many slides from Kristen Grauman and Derek Hoiem
![Page 4: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/4.jpg)
Last time
• Detecting corner-like points in an image
![Page 5: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/5.jpg)
Today
• Local invariant features– Detection of interest points
• (Harris corner detection)• Scale invariant blob detection: LoG
– Description of local patches• SIFT pipeline for invariant local features
![Page 6: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/6.jpg)
Local invariant features: outline1) Detection: Identify the interest
points
2) Description: Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
],,[ )1()1(11 dxx x
],,[ )2()2(12 dxx x
Kristen Grauman
![Page 7: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/7.jpg)
Goal: interest operator repeatability
• We want to detect (at least some of) the same points in both images.
• Yet we have to be able to run the detection procedure independently per image.
No chance to find true matches!
![Page 8: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/8.jpg)
Goal: descriptor distinctiveness
• We want to be able to reliably determine which point goes with which.
• Must provide some invariance to geometric and photometric differences between the two views.
?
![Page 9: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/9.jpg)
Local features: main components1) Detection: Identify the interest
points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
![Page 10: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/10.jpg)
Since M is symmetric, we have TXXM
2
1
0
0
iii xMx
The eigenvalues of M reveal the amount of intensity change in the two principal orthogonal gradient directions in the window.
Recall: Corners as distinctive interest points
![Page 11: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/11.jpg)
“flat” region1 and 2 are small;
“edge”:1 >> 2
2 >> 1
“corner”:1 and 2 are large, 1 ~ 2;
One way to score the cornerness:
Recall: Corners as distinctive interest points
![Page 12: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/12.jpg)
Slide: Derek Hoiem
Harris Detector [Harris88]
• Second moment matrix(autocorrelation matrix)
)()(
)()()(),( 2
2
DyDyx
DyxDxIDI III
IIIg
12
1. Image derivatives
2. Square of derivatives
3. Gaussian filter g(I)
Ix Iy
Ix2 Iy
2 IxIy
g(Ix2) g(Iy
2) g(IxIy)
222222 )]()([)]([)()( yxyxyx IgIgIIgIgIg
])),([trace()],(det[ 2DIDIhar
4. Cornerness function – both eigenvalues are strong
har5. Non-maxima suppression
1 2
1 2
det
trace
M
M
![Page 13: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/13.jpg)
Properties of the Harris corner detector
Rotation invariant?
Scale invariant?
TXXM
2
1
0
0
Yes
![Page 14: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/14.jpg)
Properties of the Harris corner detector
Rotation invariant?
Scale invariant?
All points will be classified as edges
Corner !
Yes
No
![Page 15: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/15.jpg)
Scale invariant interest points
How can we independently select interest points in each image, such that the detections are repeatable across different scales?
![Page 16: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/16.jpg)
Automatic scale selection
Intuition: •Find scale that gives local maxima of some function f in both position and scale.
f
region size
Image 1f
region size
Image 2
s1 s2
![Page 17: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/17.jpg)
What can be the “signature” function?
![Page 18: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/18.jpg)
Blob detection in 2D
Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D
2
2
2
22
y
g
x
gg
![Page 19: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/19.jpg)
Blob detection in 2D: scale selection
Laplacian-of-Gaussian = “blob” detector2
2
2
22
y
g
x
gg
fil
ter
scal
es
img1 img2 img3Bastian Leibe
![Page 20: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/20.jpg)
Blob detection in 2D
We define the characteristic scale as the scale that produces peak of Laplacian response
characteristic scale
Slide credit: Lana Lazebnik
![Page 21: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/21.jpg)
Example
Original image at ¾ the size
Kristen Grauman
![Page 22: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/22.jpg)
Original image at ¾ the size
Kristen Grauman
![Page 23: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/23.jpg)
Kristen Grauman
![Page 24: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/24.jpg)
Kristen Grauman
![Page 25: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/25.jpg)
Kristen Grauman
![Page 26: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/26.jpg)
Kristen Grauman
![Page 27: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/27.jpg)
)()( yyxx LL
List of (x, y, σ)
scale
Scale invariant interest points
Interest points are local maxima in both position and scale.
Squared filter response maps
![Page 28: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/28.jpg)
Scale-space blob detector: Example
Image credit: Lana Lazebnik
![Page 29: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/29.jpg)
We can approximate the Laplacian with a difference of Gaussians; more efficient to implement.
2 ( , , ) ( , , )xx yyL G x y G x y
( , , ) ( , , )DoG G x y k G x y
(Laplacian)
(Difference of Gaussians)
Technical detail
![Page 30: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/30.jpg)
Maximally Stable Extremal Regions [Matas ‘02]
• Based on Watershed segmentation algorithm• Select regions that stay stable over a large
parameter range
K. Grauman, B. Leibe
![Page 31: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/31.jpg)
Example Results: MSER
34 K. Grauman, B. Leibe
![Page 32: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/32.jpg)
Comparison
LoG
MSER
Harris
![Page 33: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/33.jpg)
Comparison of Keypoint Detectors
Tuytelaars Mikolajczyk 2008
![Page 34: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/34.jpg)
Choosing a detector
• What do you want it for?– Precise localization in x-y: Harris– Good localization in scale: Difference of Gaussian– Flexible region shape: MSER
• Best choice often application dependent– Harris-/Hessian-Laplace/DoG work well for many natural categories– MSER works well for buildings and printed things
• Why choose?– Get more points with more detectors
• There have been extensive evaluations/comparisons– [Mikolajczyk et al., IJCV’05, PAMI’05]– All detectors/descriptors shown here work well
![Page 35: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/35.jpg)
• For most local feature detectors, executables are available online:– http://robots.ox.ac.uk/~vgg/research/affine– http://www.cs.ubc.ca/~lowe/keypoints/– http://www.vision.ee.ethz.ch/~surf
K. Grauman, B. Leibe
![Page 36: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/36.jpg)
Local features: main components1) Detection: Identify the interest
points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
],,[ )1()1(11 dxx x
],,[ )2()2(12 dxx x
![Page 37: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/37.jpg)
Geometric transformations
e.g. scale, translation, rotation
![Page 38: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/38.jpg)
Photometric transformations
Figure from T. Tuytelaars ECCV 2006 tutorial
![Page 39: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/39.jpg)
Raw patches as local descriptors
The simplest way to describe the neighborhood around an interest point is to write down the list of intensities to form a feature vector.
But this is very sensitive to even small shifts, rotations.
![Page 40: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/40.jpg)
SIFT descriptor [Lowe 2004]
• Use histograms to bin pixels within sub-patches according to their orientation.
0 2
Why subpatches?
Why does SIFT have some illumination invariance?
![Page 41: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/41.jpg)
CSE 576: Computer Vision
Making descriptor rotation invariant
Image from Matthew Brown
• Rotate patch according to its dominant gradient orientation
• This puts the patches into a canonical orientation.
![Page 42: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/42.jpg)
• Extraordinarily robust matching technique• Can handle changes in viewpoint
• Up to about 60 degree out of plane rotation
• Can handle significant changes in illumination• Sometimes even day vs. night (below)
• Fast and efficient—can run in real time• Lots of code available
• http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT
Steve Seitz
SIFT descriptor [Lowe 2004]
![Page 43: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/43.jpg)
Example
NASA Mars Rover images
![Page 44: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/44.jpg)
NASA Mars Rover imageswith SIFT feature matchesFigure by Noah Snavely
Example
![Page 45: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/45.jpg)
SIFT properties
• Invariant to– Scale – Rotation
• Partially invariant to– Illumination changes– Camera viewpoint– Occlusion, clutter
![Page 46: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/46.jpg)
Local features: main components1) Detection: Identify the interest
points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
![Page 47: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/47.jpg)
Matching local features
Kristen Grauman
![Page 48: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/48.jpg)
Matching local features
?
To generate candidate matches, find patches that have the most similar appearance (e.g., lowest SSD)
Simplest approach: compare them all, take the closest (or closest k, or within a thresholded distance)
Image 1 Image 2
Kristen Grauman
![Page 49: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/49.jpg)
Ambiguous matches
At what SSD value do we have a good match?
To add robustness to matching, can consider ratio : distance to best match / distance to second best match
If low, first match looks good.
If high, could be ambiguous match.
Image 1 Image 2
? ? ? ?
Kristen Grauman
![Page 50: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/50.jpg)
Matching SIFT Descriptors• Nearest neighbor (Euclidean distance)• Threshold ratio of nearest to 2nd nearest descriptor
Lowe IJCV 2004
![Page 51: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/51.jpg)
Recap: robust feature-based alignment
Source: L. Lazebnik
![Page 52: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/52.jpg)
Recap: robust feature-based alignment
• Extract features
Source: L. Lazebnik
![Page 53: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/53.jpg)
Recap: robust feature-based alignment
• Extract features• Compute putative matches
Source: L. Lazebnik
![Page 54: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/54.jpg)
Recap: robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
Source: L. Lazebnik
![Page 55: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/55.jpg)
Recap: robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
• Verify transformation (search for other matches consistent with T)
Source: L. Lazebnik
![Page 56: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/56.jpg)
Recap: robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
• Verify transformation (search for other matches consistent with T)
Source: L. Lazebnik
![Page 57: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/57.jpg)
Applications of local invariant features
• Wide baseline stereo• Motion tracking• Panoramas• Mobile robot navigation• 3D reconstruction• Recognition• …
![Page 58: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/58.jpg)
Automatic mosaicing
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
![Page 59: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/59.jpg)
Wide baseline stereo
[Image from T. Tuytelaars ECCV 2006 tutorial]
![Page 60: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/60.jpg)
Recognition of specific objects, scenes
Rothganger et al. 2003 Lowe 2002
Schmid and Mohr 1997 Sivic and Zisserman, 2003
Kristen Grauman
![Page 61: Project 3 questions?](https://reader035.vdocument.in/reader035/viewer/2022062423/568149e8550346895db712f7/html5/thumbnails/61.jpg)
Summary
• Interest point detection– Harris corner detector– Laplacian of Gaussian, automatic scale selection
• Invariant descriptors– Rotation according to dominant gradient direction– Histograms for robustness to small shifts and
translations (SIFT descriptor)