Download - Locating and Describing Interest Points
![Page 1: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/1.jpg)
Locating and Describing Interest Points
Computer VisionCS 543 / ECE 549
University of Illinois
Derek Hoiem
03/02/10
Acknowledgment: Many keypoint slides from Grauman&Leibe 2008 AAAI Tutorial
![Page 2: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/2.jpg)
What is “object recognition”?
![Page 3: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/3.jpg)
1. Identify a Specific Instance• General objects
– Challenges: rotation, scale, occlusion, localization– Approaches
• Geometric configurations of keypoints (Lowe 2004)
– Works well for planar, textured objects
![Page 4: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/4.jpg)
1. Identify a Specific Instance
• Faces– Typical scenario: few examples per face,
identify or verify test example– What’s hard: changes in expression,
lighting, age, occlusion, viewpoint– Basic approaches (all nearest neighbor)
1. Project into a new subspace (or kernel space) (e.g., “Eigenfaces”=PCA)
2. Measure face features3. Make 3d face model, compare shape+appearance (e.g., AAM)
![Page 5: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/5.jpg)
2. Detect Instance of a Category • Much harder than specific instance
recognition
• Challenges– Everything in instance recognition– Intraclass variation– Representation becomes crucial
![Page 6: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/6.jpg)
2. Detect Instance of a Category
• Template or sliding window• Works well when
– Object fits well into rectangular window– Interior features are discriminative
Schneiderman Kanade 2000
![Page 7: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/7.jpg)
2. Detect Instance of a Category • Parts-based
Fischler and Elschlager 1973Felzenszwalb et al. 2008
![Page 8: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/8.jpg)
3. Assign a label to a pixel or region
• Stuff– Materials, object regions, textures, etc.– Approaches
• Label patches + CRF• Segmentation + Label Regions
![Page 9: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/9.jpg)
General Process of Object Recognition
Specify Object Model
Generate Hypotheses
Score Hypotheses
Resolution
![Page 10: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/10.jpg)
General Process of Object Recognition
Example: Template Matching
Specify Object Model
Generate Hypotheses
Score Hypotheses
Resolution
Intensity Template, at x-y
Scanning window
Normalized X-Corr
Threshold + Non-max suppression
![Page 11: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/11.jpg)
General Process of Object RecognitionExample: Keypoint-based
Instance Recognition
Specify Object Model
Generate Hypotheses
Score Hypotheses
Resolution
B1B2
B3A1
A2
A3
Affine Parameters
Choose hypothesis with max score above threshold
# Inliers
Affine-variant point locations
![Page 12: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/12.jpg)
General Process of Object RecognitionExample: Keypoint-based
Instance Recognition
Specify Object Model
Generate Hypotheses
Score Hypotheses
Resolution
B1B2
B3A1
A2
A3
Today’s Class
![Page 13: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/13.jpg)
Overview of Keypoint Matching
K. Grauman, B. Leibe
Af Bf
B1
B2
B3A1
A2 A3
Tffd BA ),(
1. Find a set of distinctive key- points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
![Page 14: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/14.jpg)
Main challenges
• Change in position and scale
• Change in viewpoint
• Occlusion
• Articulation
![Page 15: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/15.jpg)
Goals for Keypoints
Detect points that are repeatable and distinctive
![Page 16: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/16.jpg)
Key trade-offs
More Points More Repeatable
B1
B2
B3A1
A2 A3
Localization
More Robust More Selective
Description
Robust to occlusionWorks with less texture
Robust detectionPrecise localization
Deal with expected variationsMaximize correct matches
Minimize wrong matches
![Page 17: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/17.jpg)
Keypoint Localization
• Goals: – Repeatable detection– Precise localization– Interesting content
K. Grauman, B. Leibe
![Page 18: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/18.jpg)
Choosing interest points• If you wanted to meet a friend would you say
a) “Let’s meet on campus.”b) “Let’s meet on Green street.”c) “Let’s meet at Green and Wright.”– Corner detection
• Or if you were in a secluded area:a) “Let’s meet in the Plains of Akbar.”b) “Let’s meet on the side of Mt. Doom.”c) “Let’s meet on top of Mt. Doom.”– Blob (valley/peak) detection
![Page 19: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/19.jpg)
Choosing interest points
• Corners– “Let’s meet at Green and Wright.”
• Peaks/Valleys – “Let’s meet on top of Mt. Doom.”
![Page 20: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/20.jpg)
Many Existing Detectors Available
K. Grauman, B. Leibe
Hessian & Harris [Beaudet ‘78], [Harris ‘88]Laplacian, DoG [Lindeberg ‘98], [Lowe 1999]Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01]Harris-/Hessian-Affine[Mikolajczyk & Schmid ‘04]EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02]Salient Regions [Kadir & Brady ‘01] Others…
![Page 21: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/21.jpg)
Hessian Detector [Beaudet78]
• Hessian determinant
K. Grauman, B. Leibe
yyxy
xyxx
IIII
IHessian )(
Ixx
IyyIxy
Intuition: Search for strongderivatives in two orthogonal directions
![Page 22: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/22.jpg)
Hessian Detector [Beaudet78]
• Hessian determinant
K. Grauman, B. Leibe
Ixx
IyyIxy
2))(det( xyyyxx IIIIHessian
2)^(. xyyyxx III In Matlab:
yyxy
xyxx
IIII
IHessian )(
![Page 23: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/23.jpg)
Hessian Detector – Responses [Beaudet78]
Effect: Responses mainly on corners and strongly textured areas.
![Page 24: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/24.jpg)
Hessian Detector – Responses [Beaudet78]
![Page 25: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/25.jpg)
Harris Detector [Harris88]
• Second moment matrix(autocorrelation matrix)
K. Grauman, B. Leibe
)()()()(
)(),( 2
2
DyDyx
DyxDxIDI III
IIIg
Intuition: Search for local neighborhoods where the image content has two main directions (eigenvectors).
![Page 26: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/26.jpg)
Harris Detector [Harris88]
• Second moment matrix(autocorrelation matrix)
)()()()(
)(),( 2
2
DyDyx
DyxDxIDI III
IIIg
32
1. Image derivatives
2. Square of derivatives
3. Gaussian filter g(I)
Ix Iy
Ix2 Iy2 IxIy
g(Ix2) g(Iy2) g(IxIy)
222222 )]()([)]([)()( yxyxyx IgIgIIgIgIg
))],([trace()],(det[ DIDIhar 4. Cornerness function – both eigenvalues are strong
har5. Non-maxima suppression
g(IxIy)
1 2
1 2
dettrace
MM
![Page 27: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/27.jpg)
Harris Detector – Responses [Harris88]
Effect: A very precise corner detector.
![Page 28: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/28.jpg)
Harris Detector – Responses [Harris88]
![Page 29: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/29.jpg)
So far: can localize in x-y, but not scale
![Page 30: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/30.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe
)),(( )),((11
xIfxIfmm iiii
How to find corresponding patch sizes?
![Page 31: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/31.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 32: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/32.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 33: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/33.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 34: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/34.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 35: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/35.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 36: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/36.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe)),((
1xIf
mii )),((1
xIfmii
![Page 37: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/37.jpg)
What Is A Useful Signature Function?• Laplacian-of-Gaussian = “blob” detector
K. Grauman, B. Leibe
![Page 38: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/38.jpg)
Laplacian-of-Gaussian (LoG)• Local maxima in scale
space of Laplacian-of-Gaussian
K. Grauman, B. Leibe
)()( yyxx LL
List of (x, y, s)
![Page 39: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/39.jpg)
Results: Laplacian-of-Gaussian
K. Grauman, B. Leibe
![Page 40: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/40.jpg)
Difference-of-Gaussian (DoG)• Difference of Gaussians as approximation of
the Laplacian-of-Gaussian
K. Grauman, B. Leibe
- =
![Page 41: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/41.jpg)
DoG – Efficient Computation• Computation in Gaussian scale pyramid
K. Grauman, B. Leibe
Original image41
2
Sampling withstep =2
![Page 42: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/42.jpg)
Results: Lowe’s DoG
K. Grauman, B. Leibe
![Page 43: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/43.jpg)
T. Tuytelaars, B. Leibe
Orientation Normalization• Compute orientation histogram• Select dominant orientation• Normalize: rotate to fixed orientation
0 2
[Lowe, SIFT, 1999]
![Page 44: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/44.jpg)
Harris-Laplace [Mikolajczyk ‘01]
1. Initialization: Multiscale Harris corner detection
Computing Harris function Detecting local maxima
![Page 45: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/45.jpg)
Harris-Laplace [Mikolajczyk ‘01]
1. Initialization: Multiscale Harris corner detection2. Scale selection based on Laplacian
(same procedure with Hessian Hessian-Laplace)
K. Grauman, B. Leibe
Harris points
Harris-Laplace points
![Page 46: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/46.jpg)
Maximally Stable Extremal Regions [Matas ‘02]
• Based on Watershed segmentation algorithm• Select regions that stay stable over a large
parameter range
K. Grauman, B. Leibe
![Page 47: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/47.jpg)
Example Results: MSER
54 K. Grauman, B. Leibe
![Page 48: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/48.jpg)
Available at a web site near you…• For most local feature detectors, executables
are available online:– http://robots.ox.ac.uk/~vgg/research/affine– http://www.cs.ubc.ca/~lowe/keypoints/– http://www.vision.ee.ethz.ch/~surf
K. Grauman, B. Leibe
![Page 49: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/49.jpg)
Local Descriptors• The ideal descriptor should be
– Robust– Distinctive– Compact– Efficient
• Most available descriptors focus on edge/gradient information– Capture texture information– Color rarely used
K. Grauman, B. Leibe
![Page 50: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/50.jpg)
Local Descriptors: SIFT Descriptor
[Lowe, ICCV 1999]
Histogram of oriented gradients• Captures important texture information• Robust to small translations / affine deformations
K. Grauman, B. Leibe
![Page 51: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/51.jpg)
Details of Lowe’s SIFT algorithm• Run DoG detector
– Find maxima in location/scale space– Remove edge points
• Find all major orientations– Bin orientations into 36 bin histogram
• Weight by gradient magnitude• Weight by distance to center (Gaussian-weighted mean)
– Return orientations within 0.8 of peak• Use parabola for better orientation fit
• For each (x,y,scale,orientation), create descriptor:– Sample 16x16 gradient mag. and rel. orientation– Bin 4x4 samples into 4x4 histograms– Threshold values to max of 0.2, divide by L2 norm– Final descriptor: 4x4x8 normalized histograms
Lowe IJCV 2004
![Page 52: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/52.jpg)
Matching SIFT Descriptors• Nearest neighbor (Euclidean distance)• Threshold ratio of nearest to 2nd nearest descriptor
Lowe IJCV 2004
![Page 53: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/53.jpg)
SIFT Repeatability
Lowe IJCV 2004
![Page 54: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/54.jpg)
SIFT Repeatability
![Page 55: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/55.jpg)
SIFT Repeatability
Lowe IJCV 2004
![Page 56: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/56.jpg)
Local Descriptors: SURF
K. Grauman, B. Leibe
• Fast approximation of SIFT idea Efficient computation by 2D box filters &
integral images 6 times faster than SIFT
Equivalent quality for object identification
[Bay, ECCV’06], [Cornelis, CVGPU’08]
• GPU implementation available Feature extraction @ 200Hz
(detector + descriptor, 640×480 img) http://www.vision.ee.ethz.ch/~surf
![Page 57: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/57.jpg)
Local Descriptors: Shape Context
Count the number of points inside each bin, e.g.:
Count = 4
Count = 10...
Log-polar binning: more precision for nearby points, more flexibility for farther points.
Belongie & Malik, ICCV 2001K. Grauman, B. Leibe
![Page 58: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/58.jpg)
Local Descriptors: Geometric Blur
Example descriptor
~
Compute edges at four orientations
Extract a patchin each channel
Apply spatially varyingblur and sub-sample
(Idealized signal)
Berg & Malik, CVPR 2001K. Grauman, B. Leibe
![Page 59: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/59.jpg)
Choosing a detector
• What do you want it for?– Precise localization in x-y: Harris– Good localization in scale: Difference of Gaussian– Flexible region shape: MSER
• Best choice often application dependent– Harris-/Hessian-Laplace/DoG work well for many natural categories– MSER works well for buildings and printed things
• Why choose?– Get more points with more detectors
• There have been extensive evaluations/comparisons– [Mikolajczyk et al., IJCV’05, PAMI’05]– All detectors/descriptors shown here work well
![Page 60: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/60.jpg)
Comparison of Keypoint Detectors
Tuytelaars Mikolajczyk 2008
![Page 61: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/61.jpg)
Choosing a descriptor
• Again, need not stick to one
• For object instance recognition or stitching, SIFT or variant is a good choice
![Page 62: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/62.jpg)
Things to remember
• Keypoint detection: repeatable and distinctive– Corners, blobs, stable regions– Harris, DoG
• Descriptors: robust and selective– spatial histograms of orientation– SIFT
![Page 63: Locating and Describing Interest Points](https://reader033.vdocument.in/reader033/viewer/2022051702/568149e8550346895db712d3/html5/thumbnails/63.jpg)
Next time
• Recognizing objects using keypoints