interest points ii
DESCRIPTION
Interest Points II. TODAY : Invariant Features, SIFT Recap: Interest Points and Descriptors Harris, Correlation Image Sampling Scaling, rotation SIFT (scale invariant feature transform) Both interest point detector (DOG)… …and descriptor. Interest Points and Descriptors. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/1.jpg)
04/22/23 1
TODAY: Invariant Features, SIFT Recap: Interest Points and Descriptors
Harris, Correlation Image Sampling
Scaling, rotation SIFT
(scale invariant feature transform) Both interest point detector (DOG)… …and descriptor
Interest Points II
![Page 2: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/2.jpg)
04/22/23 2
Interest Points Focus attention for image understanding Points with 2D structure in the image Examples: Harris corners, DOG maxima
Descriptors Describe region around an interest point Must be invariant under
Geometric distortion Photometric changes
Examples: Oriented patches, SIFT
Interest Points and Descriptors
![Page 3: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/3.jpg)
04/22/23 3
Image Sampling : Scale
![Page 4: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/4.jpg)
04/22/23 4
Image Sampling : ScaleThis is aliasing!
![Page 5: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/5.jpg)
04/22/23 5
Recall formula for 2D rotation:
Given a 2D grid of x, y pixel values… what are the u, v pixel values?
Image Sampling : Rotation
U
V
X
Y
θ
1
1
??
??
![Page 6: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/6.jpg)
04/22/23 6
Image Sampling : Rotation
Problem: sample grids are not aligned…
U, V
X, Y
![Page 7: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/7.jpg)
04/22/23 7
?
?
?
Linear Interpolation
x1
???
I
I1
I2
Find value between pixel samples
I00
I10
I01 I11I
x
y
![Page 8: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/8.jpg)
04/22/23 8
Bilinear Interpolation
Use all 4 adjacent samples
x
y
I00 I10
I01 I11
![Page 9: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/9.jpg)
The SIFT (Scale Invariant Feature Transform) Detector and Descriptor
developed by David LoweUniversity of British ColumbiaInitial paper 1999Newer journal paper 2004
![Page 10: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/10.jpg)
04/22/23 10
Motivation
The Harris operator is not invariant to scale and its descriptor was not invariant to rotation1.
For better image matching, Lowe’s goal was to develop an operator that is invariant to scale and rotation.
The operator he developed is both a detector and a descriptor and can be used for both image matching and object recognition.
1But Schmid and Mohr developed a rotation invariant descriptor for it in 1997.
![Page 11: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/11.jpg)
04/22/23 11
Idea of SIFT Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters
SIFT Features
![Page 12: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/12.jpg)
04/22/23 12
Claimed Advantages of SIFT
Locality: features are local, so robust to occlusion and clutter (no prior segmentation)
Distinctiveness: individual features can be matched to a large database of objects
Quantity: many features can be generated for even small objects
Efficiency: close to real-time performance Extensibility: can easily be extended to wide range
of differing feature types, with each adding robustness
![Page 13: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/13.jpg)
04/22/23 13
Overall Procedure at a High Level1. Scale-space extrema detection
2. Keypoint localization
3. Orientation assignment
4. Keypoint description
Search over multiple scales and image locations.
Fit a model to detrmine location and scale.Select keypoints based on a measure of stability.
Compute best orientation(s) for each keypoint region.
Use local image gradients at selected scale and rotationto describe each keypoint region.
![Page 14: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/14.jpg)
04/22/23 14
1. Scale-space extrema detection Goal: Identify locations and scales that can be repeatably assigned under different views of the same scene or object.
Method: search for stable features across multiple scales using a continuous function of scale.
Prior work has shown that under a variety of assumptions, the best function is a Gaussian function.
The scale space of an image is a function L(x,y,) that is produced from the convolution of a Gaussian kernel (at different scales) with the input image.
![Page 15: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/15.jpg)
04/22/23 15
Aside: Image Pyramids
Bottom level is the original image.
2nd level is derived from theoriginal image according tosome function
3rd level is derived from the2nd level according to the samefuntion
And so on.
![Page 16: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/16.jpg)
04/22/23 16
Aside: Mean Pyramid
Bottom level is the original image.
At 2nd level, each pixel is the meanof 4 pixels in the original image.
At 3rd level, each pixel is the meanof 4 pixels in the 2nd level.
And so on.
mean
![Page 17: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/17.jpg)
04/22/23 17
Aside: Gaussian PyramidAt each level, image is smoothed and reduced in size.
Bottom level is the original image.
At 2nd level, each pixel is the resultof applying a Gaussian mask tothe first level and then subsamplingto reduce the size.
And so on.
Apply Gaussian filter
![Page 18: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/18.jpg)
04/22/23 18
Example: Subsampling with Gaussian pre-filtering
G 1/4
G 1/8
Gaussian 1/2
![Page 19: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/19.jpg)
04/22/23 19
Lowe’s Scale-space Interest Points Laplacian of Gaussian kernel
Scale normalised (x by scale2) Proposed by Lindeberg
Scale-space detection Find local maxima across scale/space A good “blob” detector
![Page 20: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/20.jpg)
04/22/23 20
Lowe’s Scale-space Interest Points
Gaussian is an ad hoc solution of heat diffusion equation
Hence
k is not necessarily very small in practice
![Page 21: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/21.jpg)
04/22/23 21
Lowe’s Scale-space Interest Points Scale-space function L
Gaussian convolution
Difference of Gaussian kernel is a close approximate to scale-normalized Laplacian of Gaussian
Can approximate the Laplacian of Gaussian kernel with a difference of separable convolutions
where is the width of the Gaussian.
2 scales: and k
![Page 22: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/22.jpg)
04/22/23 22
Lowe’s Pyramid Scheme• Scale space is separated into octaves:
• Octave 1 uses scale • Octave 2 uses scale 2• etc.
• In each octave, the initial image is repeatedly convolved with Gaussians to produce a set of scale space images.
• Adjacent Gaussians are subtracted to produce the DOG
• After each octave, the Gaussian image is down-sampled by a factor of 2 to produce an image ¼ the size to start the next level.
![Page 23: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/23.jpg)
04/22/23 23
Lowe’s Pyramid Scheme
s+2 filterss+1=2(s+1)/s0
.
.i=2i/s0
.
.2=22/s0
1=21/s0
0
s+3imagesincludingoriginal
s+2differ-enceimages
The parameter s determines the number of images per octave.
![Page 24: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/24.jpg)
04/22/23 24
Key point localization
Detect maxima and minima of difference-of-Gaussian in scale space
Each point is compared to its 8 neighbors in the current image and 9 neighbors each in the scales above and below
Blur
Resamp le
Subtract
For each max or min found,output is the location andthe scale.
s+2 difference images.top and bottom ignored.s planes searched.
![Page 25: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/25.jpg)
04/22/23 25
Scale-space extrema detection: experimental results over 32 images that were synthetically transformed and noise added.
Sampling in scale for efficiency How many scales should be used per octave? S=?
More scales evaluated, more keypoints found S < 3, stable keypoints increased too S > 3, stable keypoints decreased S = 3, maximum stable keypoints found
% detected
% correctly matched
average no. detected
average no. matched
Stability Expense
![Page 26: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/26.jpg)
04/22/23 26
2. Keypoint localization Detailed keypoint determination
Sub-pixel and sub-scale location scale determination
Ratio of principal curvature to reject edges and flats (like detecting corners)
![Page 27: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/27.jpg)
04/22/23 27
Keypoint localization Once a keypoint candidate is found, perform a detail
ed fit to nearby data to determine location, scale, and ratio of principal curvatures
In initial work keypoints were found at location and scale of a central sample point.
In newer work, they fit a 3D quadratic function to improve interpolation accuracy.
The Hessian matrix was used to eliminate edge responses.
![Page 28: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/28.jpg)
04/22/23 28
Eliminating the Edge Response Reject flats:
< 0.03 Reject edges:
r < 10 What does this look like?
Let be the eigenvalue withlarger magnitude and the smaller.
Let r = /.So = r
(r+1)2/r is at amin when the2 eigenvaluesare equal.
![Page 29: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/29.jpg)
04/22/23 29
3. Orientation assignment
Create histogram of local gradient directions at selected scale
Assign canonical orientation at peak of smoothed histogram
Each key specifies stable 2D coordinates (x, y, scale,orientation)
If 2 major orientations, use both.
![Page 30: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/30.jpg)
04/22/23 30
Keypoint localization with orientation832
729536
233x189initial keypoints
keypoints aftergradient threshold
keypoints afterratio threshold
![Page 31: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/31.jpg)
04/22/23 31
4. Keypoint Descriptors
At this point, each keypoint has location scale orientation
Next is to compute a descriptor for the local image region about each keypoint that is highly distinctive invariant as possible to variations such as
changes in viewpoint and illumination
![Page 32: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/32.jpg)
04/22/23 32
Normalization
Rotate the window to standard orientation
Scale the window size based on the scale at which the point was found.
![Page 33: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/33.jpg)
04/22/23 33
Lowe’s Keypoint Descriptor
use the normalized circular region about the keypoint
compute gradient magnitude and orientation at each point in the region
weight them by a Gaussian window overlaid on the circle
create an orientation histogram over the 4 X 4 subregions of the window
4 X 4 descriptors over 16 X 16 sample array were used in practice. 4 X 4 times 8 directions gives a vector of 128 values.
![Page 34: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/34.jpg)
04/22/23 34
Lowe’s Keypoint Descriptor(shown with 2 X 2 descriptors over 8 X 8) Invariant to other changes (Complex Cell)
In experiments, 4x4 arrays of 8 bin histogram is used, a total of 128 features for one keypoint
![Page 35: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/35.jpg)
04/22/23 35
Scale Invariant Detectors Harris-Laplacian1
Find local maximum of: Harris corner detector in space
(image coordinates) Laplacian in scale
1 K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 20012 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004
scale
x
y
Harris L
apla
cian
• SIFT (Lowe)2
Find local maximum of:
– Difference of Gaussians in space and scale
scale
x
y
DoG
D
oG
![Page 36: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/36.jpg)
04/22/23 36
Scale Invariant Detectors
K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001
Experimental evaluation of detectors w.r.t. scale change
Repeatability rate:# correspondences
# possible correspondences
![Page 37: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/37.jpg)
04/22/23 37
Schmid’s Comparison with Harris-Laplacian Affine-invariant comparison
Translation-invariant – local features: both OK Rotation-invariant
Harris-Laplacian PCA
SIFT Orientation
Shear-invariant Harris-Laplacian
Eigenvalues SIFT
No Within 50 degree of viewpoint, SIFT is better than H
L, after 70 degree, HL is better.
![Page 38: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/38.jpg)
04/22/23 38
Comparison with Harris-Laplacian Computational time:
SIFT uses few floating point calculation HL uses iterative calculation which costs much
more
![Page 39: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/39.jpg)
04/22/23 39
Using SIFT for Matching “Objects”
![Page 40: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/40.jpg)
04/22/23 40
![Page 41: Interest Points II](https://reader036.vdocument.in/reader036/viewer/2022081513/56816864550346895ddeb84e/html5/thumbnails/41.jpg)
04/22/23 41
Uses for SIFT
Feature points are used also for: Image alignment (homography, fundamental matr
ix) 3D reconstruction Motion tracking Object recognition Indexing and database retrieval Robot navigation … other