scale-space image processing - stanford university · (a) multiple extrema in log scale space can...
TRANSCRIPT
![Page 1: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/1.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 1
Scale-space image processing Corresponding image features can appear at different scales
Like shift-invariance, scale-invariance of image processing algorithms is often desirable.
Scale-space representation is useful to process an image in a manner that is both shift-invariant and scale-invariant
![Page 2: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/2.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 2
Scale-space image processing
Scale-space theory Laplacian of Gaussian (LoG) and Difference of Gaussian (DoG) Scale-space edge detection Scale-space keypoint detection
Harris-Laplacian SIFT detector SURF detector
![Page 3: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/3.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 3
Scale-space representation of a signal
Successive smoothing with a Gaussian filter
Zero-crossings of 2nd derivative Fewer edges at coarser scales
Parametric family of signals f t (x) where fine-scale information is successively attenuated
scale t
ft′′ x( )
![Page 4: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/4.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 4
Scale-space representation of images
Parametric family of images smoothed by Gaussian filter
Shift-invariance
Rotation-invariance
Original image f (x,y)
Coarser scales t
![Page 5: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/5.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 5
Scale-space representation of images (cont.)
Commutative semigroup property
Separability
![Page 6: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/6.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 6
Scale-space representation of images (cont.)
Non-creation of local extrema (for f (x,y) and all of its partial derivatives) since and unimodal.
Solution to diffusion equation (heat equation)
![Page 7: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/7.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 7
![Page 8: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/8.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 8
LoG vs. DoG
-4-2
02
4
-4
-2
0
2
4
-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
XY -4-2
02
4
-4
-2
0
2
4
-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
XY
Laplacian of Gaussian Difference of Gaussians t = σ2 = 1 t = σ2 = 1, k = 1.1
12∇2 f t x, y( )= ∂
∂tf t x, y( )
![Page 9: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/9.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 9
LoG vs. DoG (cont.) Laplacian of Gaussian Difference of Gaussians
-20
2
-2
0
2
0
0.2
0.4
0.6
0.8
ωxωy
|H|
-20
2
-2
0
2
0
0.2
0.4
0.6
0.8
ωxωy
|H|
t = σ2 = 1 t = σ2 = 1, k = 1.1
![Page 10: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/10.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 10
Scale space: Laplacian images
t = 1 t = 4 t = 16 t = 64
![Page 11: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/11.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 11
Scale space: Binarized Laplacian images
t = 1 t = 4 t = 16 t = 64
![Page 12: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/12.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 12
Scale space: edge detection Zero crossings of Laplacian images
Low-gradient-magnitude edges removed
t = 1 t = 4 t = 16 t = 64
![Page 13: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/13.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 13
Laplacian zero-crossings
![Page 14: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/14.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 14
???
Suppose we filter with the following 3x3 Laplacian kernel. How many arithmetic operations per pixel are required? Alternatively, suppose we have and and we compute the difference to approximate the LoG by a DoG. How much do we save relative to the arithmetic operations required for the LoG computation?
f2t( )[x, y]− f t[x, y]
0 1 01 −4 1
0 1 0
f2t( )[x, y]
![Page 15: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/15.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 15
Scale-space representation provides all scales; which scale is best for keypoint detection?
Harris-Laplacian 1. Detect Harris corners at some initial scale 2. For each Harris corner
detect characteristic scale
3. Apply Harris detector in a spatial neighborhood at scale to refine keypoint location
4. Repeat 2. and 3. until convergence
th = argmax
tt ⋅∇2 f t xh , yh( )
th
scale t
x
y Harris
Harris
Harris
Keypoint detection with automatic scale selection
![Page 16: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/16.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 16
Harris-Laplacian example (150 strongest peaks)
Keypoint detection with automatic scale selection
![Page 17: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/17.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 17
Keypoint detection with automatic scale selection Harris-Laplacian example (200 strongest peaks)
![Page 18: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/18.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 18
SIFT - Scale-Invariant Feature Transform Decompose image into DoG scale-space representation Detect minima and maxima locally and across scales Fit 3-d quadratic function to localize extrema with sub-
pixel/sub-scale accuracy [Brown, Lowe, 2002] Eliminate edge responses based on Hessian
t = 1
t = 2
t = 2
t = 2 2
t = 4
t = 4
t = 4 2
t = 8
t = 8 2
t = 16
Gaussian Difference of
Gaussian (DoG)
Scale (first
octave)
Scale (next
octave)
Scale
… SIFT keypoint detection
[Lowe, 1999, 2004]
![Page 19: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/19.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 19
SIFT scale space pyramid: octave 1
-
-
-
![Page 20: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/20.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 20
SIFT scale space pyramid: octave 2
-
-
-
![Page 21: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/21.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 21
SIFT scale space pyramid: octave 3
-
-
-
![Page 22: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/22.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 22
SIFT scale space pyramid: octave 4
-
-
-
![Page 23: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/23.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 23
SIFT scale space pyramid: octave 5
-
-
-
![Page 24: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/24.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 24
SIFT keypoints
![Page 25: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/25.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 25
SIFT keypoints
![Page 26: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/26.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 26
Robustness against scaling
[Mikolajczyk, Schmid, 2001]
![Page 27: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/27.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 27
???
Which of the following statements are true? (a) Local extrema in LoG scale space can occur along edges. (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at
different scales.
(b) The SIFT keypoint detector only detects dark blobs in an image. The detector must be applied to a negative of the image to detect bright blobs.
(d) The SIFT keypoint detector finds more keypoints at small scales than at large scales in a typical image.
![Page 28: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/28.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 28
Hessian keypoints in scale space
![Page 29: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/29.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 29
SURF keypoint detection
SURF – Speeded Up Robust Features [Bay, Tuytelaars, Van Gool, ECCV 2006]
No subsampling – all resolution levels at full spatial resolution Simple approximation of scale space Gaussian derivatives using integral images
Determinant of Hessian
Non-maximum suppression in 3x3x3 [x,y,t] neighborhood
Interpolation of maximum of det(H) in image space x,y and scale t
Dt
xy
![Page 30: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/30.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 30
SURF keypoints
![Page 31: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/31.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 31
SIFT keypoints
![Page 32: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/32.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 32
SURF keypoints
![Page 33: Scale-space image processing - Stanford University · (a) Multiple extrema in LoG scale space can occur at the same [x,y] location, but at different scales. (b) The SIFT keypoint](https://reader035.vdocument.in/reader035/viewer/2022070110/604722852ddc9a0bc0170ffa/html5/thumbnails/33.jpg)
Digital Image Processing: Bernd Girod, © 2013 Stanford University -- Scale Space 33
SIFT keypoints