features - 國立臺灣大學cyy/courses/vfx/19... · harris corner detector measure of corner...
TRANSCRIPT
![Page 1: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/1.jpg)
Features
Digital Visual EffectsYung-Yu Chuang
with slides by Trevor Darrell Cordelia Schmid, David Lowe, Darya Frolova, Denis Simakov, Robert Collins and Jiwon Kim
![Page 2: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/2.jpg)
Outline
• Features• Harris corner detector• SIFT• Extensions• Applications
![Page 3: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/3.jpg)
Features
![Page 4: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/4.jpg)
Features
• Also known as interesting points, salient points or keypoints. Points that you can easily point out their correspondences in multiple images using only local information.
?
![Page 5: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/5.jpg)
Desired properties for features
• Distinctive: a single feature can be correctly matched with high probability.
• Invariant: invariant to scale, rotation, affine, illumination and noise for robust matching across a substantial range of affine distortion, viewpoint change and so on. That is, it is repeatable.
![Page 6: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/6.jpg)
Applications
• Object or scene recognition• Structure from motion• Stereo• Motion tracking• …
![Page 7: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/7.jpg)
Components
• Feature detection locates where they are• Feature description describes what they are• Feature matching decides whether two are the
same one
![Page 8: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/8.jpg)
Harris corner detector
![Page 9: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/9.jpg)
Moravec corner detector (1980)
• We should easily recognize the point by looking through a small window
• Shifting a window in any direction should give a large change in intensity
![Page 10: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/10.jpg)
Moravec corner detector
flat
![Page 11: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/11.jpg)
Moravec corner detector
flat
![Page 12: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/12.jpg)
Moravec corner detector
flat edge
![Page 13: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/13.jpg)
Moravec corner detector
flat edge cornerisolated point
![Page 14: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/14.jpg)
yx
yxIvyuxIyxwvuE,
2),(),(),(),(
Moravec corner detector
Change of intensity for the shift [u,v]:
window function
Four shifts: (u,v) = (1,0), (1,1), (0,1), (-1, 1)Look for local maxima in min{E}
intensityshifted intensity
![Page 15: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/15.jpg)
Problems of Moravec detector
• Noisy response due to a binary window function• Only a set of shifts at every 45 degree is
considered• Only minimum of E is taken into account
Harris corner detector (1988) solves these problems.
![Page 16: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/16.jpg)
Harris corner detector
Noisy response due to a binary window functionUse a Gaussian function
![Page 17: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/17.jpg)
Harris corner detector
Only a set of shifts at every 45 degree is consideredConsider all small shifts by Taylor’s expansion
![Page 18: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/18.jpg)
Harris corner detector
Only a set of shifts at every 45 degree is consideredConsider all small shifts by Taylor’s expansion
yxyx
yxy
yxx
yxIyxIyxwC
yxIyxwB
yxIyxwABvCuvAuvuE
,
,
2
,
2
22
),(),(),(
),(),(
),(),(2),(
yx
yxIvyuxIyxwvuE,
2),(),(),(),(
yx
yx vuOvIuIyxw,
222 ),(),(
![Page 19: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/19.jpg)
Harris corner detector
Equivalently, for small shifts [u,v] we have a bilinearapproximation:
, where M is a 22 matrix computed from image derivatives:
vu
vuvuE ),( M
yx yyx
yxx
IIIIII
yxw,
2
2
),(M
![Page 20: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/20.jpg)
Harris corner detector (matrix form)
Muu
uxx
u
ux
ux
xux 00
T
TT
T
T
II
I
III
II
2
2
00
2|)()(|
)(
2|)()(|)()(px
0000
xuxxuW
IIwE
![Page 21: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/21.jpg)
Harris corner detector
Only minimum of E is taken into accountA new corner measurement by investigating the shape of the error function
represents a quadratic function; Thus, we can analyze E’s shape by looking at the property of M
MuuT
![Page 22: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/22.jpg)
Harris corner detector
High-level idea: what shape of the error function will we prefer for features?
02
46
810
12
0
5
10
0
20
40
60
80
100
02
46
810
12
0
5
10
0
20
40
60
80
100
02
46
810
12
0
5
10
0
20
40
60
80
100
flat edge corner
![Page 23: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/23.jpg)
Quadratic forms
• Quadratic form (homogeneous polynomial of degree two) of n variables xi
• Examples
=
![Page 24: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/24.jpg)
Symmetric matrices
• Quadratic forms can be represented by a real symmetric matrix A where
![Page 25: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/25.jpg)
Eigenvalues of symmetric matrices
Brad Osgood
![Page 26: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/26.jpg)
Eigenvectors of symmetric matrices
![Page 27: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/27.jpg)
Eigenvectors of symmetric matrices
zz
yΛyΛ
ΛyyxQΛxQ
xQΛQxAxx
T
T
T
TTT
TT
T
21
21
11q22 q
1 xxT
1zzT
![Page 28: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/28.jpg)
Harris corner detector
Intensity change in shifting window: eigenvalue analysis
1, 2 – eigenvalues of M
direction of the slowest change
direction of the fastest change
(max)-1/2
(min)-1/2
Ellipse E(u,v) = const
vu
vuvuE ,),( M
![Page 29: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/29.jpg)
Visualize quadratic functionsT
1001
1001
1001
1001
A
![Page 30: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/30.jpg)
Visualize quadratic functionsT
1001
1004
1001
1004
A
![Page 31: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/31.jpg)
Visualize quadratic functions
T
50.087.087.050.0
4001
50.087.087.050.0
75.130.130.125.3
A
![Page 32: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/32.jpg)
Visualize quadratic functions
T
50.087.087.050.0
10001
50.087.087.050.0
25.390.390.375.7
A
![Page 33: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/33.jpg)
Harris corner detector
1
2
Corner1 and 2 are large,1 ~ 2;E increases in all directions
1 and 2 are small;E is almost constant in all directions
edge 1 >> 2
edge 2 >> 1
flat
Classification of image points using eigenvalues of M:
![Page 34: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/34.jpg)
Harris corner detector
Measure of corner response:
(k – empirical constant, k = 0.04-0.06)
24)( 0110
211001100 aaaaaa
2tracedet MM kR
21
21
tracedet
MM
Only for reference, you do not need them to compute R
![Page 35: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/35.jpg)
Harris corner detector
![Page 36: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/36.jpg)
Another view
![Page 37: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/37.jpg)
Another view
![Page 38: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/38.jpg)
Another view
![Page 39: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/39.jpg)
Summary of Harris detector
1. Compute x and y derivatives of image
2. Compute products of derivatives at every pixel
3. Compute the sums of the products of derivatives at each pixel
IGI xx IGI y
y
xxx III 2 yyyIII 2 yxxy III
22 ' xx IGS 22 ' yyIGS xyxy IGS '
![Page 40: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/40.jpg)
Summary of Harris detector
4. Define the matrix at each pixel
5. Compute the response of the detector at each pixel
6. Threshold on value of R; compute nonmax suppression.
),(),(
),(),(),(
2
2
yxSyxSyxSyxS
yxMyxy
xyx
2tracedet MkMR
![Page 41: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/41.jpg)
Harris corner detector (input)
![Page 42: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/42.jpg)
Corner response R
![Page 43: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/43.jpg)
Threshold on R
![Page 44: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/44.jpg)
Local maximum of R
![Page 45: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/45.jpg)
Harris corner detector
![Page 46: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/46.jpg)
Harris detector: summary
• Average intensity change in direction [u,v] can be expressed as a bilinear form:
• Describe a point in terms of eigenvalues of M:measure of corner response
• A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive
vu
vuvuE ,),( M
22121 kR
![Page 47: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/47.jpg)
Now we know where features are
• But, how to match them?• What is the descriptor for a feature? The
simplest solution is the intensities of its spatial neighbors. This might not be robust to brightness change or small shift/rotation.
( )
1 2 3
4 5 6
7 8 9
1 2 3 4 5 6 7 8 9
![Page 48: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/48.jpg)
Harris detector: some properties• Partial invariance to affine intensity change
Only derivatives are used => invariance to intensity shift I I + b
Intensity scale: I a I
R
x (image coordinate)
threshold
R
x (image coordinate)
![Page 49: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/49.jpg)
Harris Detector: Some Properties
• Rotation invariance
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response R is invariant to image rotation
![Page 50: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/50.jpg)
Harris Detector is rotation invariant
Repeatability rate:# correspondences
# possible correspondences
![Page 51: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/51.jpg)
Harris Detector: Some Properties
• But: not invariant to image scale!
All points will be classified as edges
Corner !
![Page 52: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/52.jpg)
Harris detector: some properties
• Quality of Harris detector for different scale changes
Repeatability rate:# correspondences
# possible correspondences
![Page 53: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/53.jpg)
Scale invariant detection
• Consider regions (e.g. circles) of different sizes around a point
• Regions of corresponding sizes will look the same in both images
![Page 54: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/54.jpg)
Scale invariant detection
• The problem: how do we choose corresponding circles independently in each image?
• Aperture problem
![Page 55: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/55.jpg)
SIFT (Scale Invariant Feature Transform)
![Page 56: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/56.jpg)
SIFT• SIFT is an carefully designed procedure with
empirically determined parameters for the invariant and distinctive features.
![Page 57: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/57.jpg)
SIFT stages:
• Scale-space extrema detection• Keypoint localization• Orientation assignment• Keypoint descriptor
( )local descriptor
detector
descriptor
A 500x500 image gives about 2000 features
![Page 58: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/58.jpg)
1. Detection of scale-space extrema
• For scale invariance, search for stable features across all possible scales using a continuous function of scale, scale space.
• SIFT uses DoG filter for scale space because it is efficient and as stable as scale-normalized Laplacian of Gaussian.
![Page 59: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/59.jpg)
DoG filtering
Convolution with a variable-scale Gaussian
Difference-of-Gaussian (DoG) filter
Convolution with the DoG filter
![Page 60: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/60.jpg)
Scale space doubles for the next octave
K=2(1/s)
Dividing into octave is for efficiency only.
![Page 61: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/61.jpg)
Detection of scale-space extrema
![Page 62: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/62.jpg)
Keypoint localization
X is selected if it is larger or smaller than all 26 neighbors
![Page 63: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/63.jpg)
Decide scale sampling frequency
• It is impossible to sample the whole space, tradeoff efficiency with completeness.
• Decide the best sampling frequency by experimenting on 32 real image subject to synthetic transformations. (rotation, scaling, affine stretch, brightness and contrast change, adding noise…)
![Page 64: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/64.jpg)
Decide scale sampling frequency
![Page 65: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/65.jpg)
Decide scale sampling frequency
s=3 is the best, for larger s, too many unstable features
for detector, repeatability
for descriptor, distinctiveness
![Page 66: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/66.jpg)
Pre-smoothing
=1.6, plus a double expansion
![Page 67: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/67.jpg)
Scale invariance
![Page 68: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/68.jpg)
2. Accurate keypoint localization
• Reject points with low contrast (flat) and poorly localized along an edge (edge)
• Fit a 3D quadratic function for sub-pixel maxima
1
65
0-1 +1
![Page 69: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/69.jpg)
2. Accurate keypoint localization
• Reject points with low contrast (flat) and poorly localized along an edge (edge)
• Fit a 3D quadratic function for sub-pixel maxima
1
65
0-1 +1
2
2)0('')0(')0()( xfxffxf
31ˆ x
22 3262626)( xxxxxf
062)(' xxf
316
313
3126)ˆ(
2
xf
316
31
![Page 70: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/70.jpg)
2. Accurate keypoint localization
• Taylor series of several variables
• Two variables
222
22
221)0,0(),( y
yyfxy
yxfx
xxfy
yfx
xffyxf
yx
yyf
yxf
yxf
xxf
yxyx
yf
xff
yx
f 22
22
21
00
xx
xxx
0x 2
2
21
ffff T
T
![Page 71: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/71.jpg)
Accurate keypoint localization
• Taylor expansion in a matrix form, x is a vector, f maps x to a scalar
nxf
xfxf
1
1
2
2
2
2
1
2
2
2
22
2
12
21
2
21
2
21
2
nnn
n
n
xf
xxf
xxf
xxf
xf
xxf
xxf
xxf
xf
Hessian matrix(often symmetric)
gradient
![Page 72: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/72.jpg)
2D illustration
![Page 73: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/73.jpg)
2D example
-17 -1 -1
7
77
7
-9
-9
![Page 74: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/74.jpg)
Derivation of matrix form
xgx T)(h xh
![Page 75: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/75.jpg)
Derivation of matrix form
xgx T)(h
n
n
x
xgg
1
1
n
iii xg
1
gx
nn
g
g
xh
xh
h11
![Page 76: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/76.jpg)
Derivation of matrix form
Axxx T)(h
xh
![Page 77: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/77.jpg)
Derivation of matrix form
Axxx T)(h
nnnn
n
n
x
x
aa
aaxx
1
1
111
1
n
i
n
jjiij xxa
1 1
AxxAx
T
n
i
n
jjnjiin
n
i
n
jjjii
n
xaxa
xaxa
xh
xh
h
1 1
1 111
1
xAAT )(
![Page 78: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/78.jpg)
Derivation of matrix form
xffxffff T
2
2
2
2
2
2
21
xxxxxx
![Page 79: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/79.jpg)
Accurate keypoint localization
• x is a 3-vector• Change sample point if offset is larger than 0.5• Throw out low contrast (<0.03)
![Page 80: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/80.jpg)
Accurate keypoint localization
• Throw out low contrast 03.0|)ˆ(| xD
xx
xx
xx
xxxx
x
xxxxxx
x
xxxxxx
x
xx
xxx
x
ˆ21
)ˆ(21ˆ
21ˆ
21ˆ
21ˆ
ˆˆ21ˆ)ˆ(
1
2
2
1
2
2
2
2
2
2
1
2
2
2
21
2
2
2
2
T
TT
TT
TTT
TT
TT
DD
DDD
DDDDD
DDDDDDD
DDDDDDD
DDDD
![Page 81: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/81.jpg)
Eliminating edge responses
r=10
Let
Keep the points with
Hessian matrix at keypoint location
![Page 82: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/82.jpg)
Maxima in D
![Page 83: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/83.jpg)
Remove low contrast and edges
![Page 84: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/84.jpg)
Keypoint detector
233x89 832 extrema
729 after con-trast filtering
536 after cur-vature filtering
![Page 85: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/85.jpg)
3. Orientation assignment
• By assigning a consistent orientation, the keypoint descriptor can be orientation invariant.
• For a keypoint, L is the Gaussian-smoothed image with the closest scale,
orientation histogram (36 bins)
(Lx, Ly)
m
θ
![Page 86: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/86.jpg)
Orientation assignment
![Page 87: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/87.jpg)
Orientation assignment
![Page 88: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/88.jpg)
Orientation assignment
![Page 89: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/89.jpg)
Orientation assignment
σ=1.5*scale of the keypoint
![Page 90: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/90.jpg)
Orientation assignment
![Page 91: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/91.jpg)
Orientation assignment
![Page 92: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/92.jpg)
Orientation assignmentaccurate peak position is determined by fitting
![Page 93: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/93.jpg)
Orientation assignment
0 2
36-bin orientation histogram over 360˚, weighted by m and 1.5*scale falloffPeak is the orientationLocal peak within 80% creates multiple
orientationsAbout 15% has multiple orientations
and they contribute a lot to stability
![Page 94: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/94.jpg)
SIFT descriptor
![Page 95: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/95.jpg)
4. Local image descriptor• Thresholded image gradients are sampled over 16x16
array of locations in scale space• Create array of orientation histograms (w.r.t. key
orientation)• 8 orientations x 4x4 histogram array = 128 dimensions• Normalized, clip values larger than 0.2, renormalize
σ=0.5*width
![Page 96: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/96.jpg)
Why 4x4x8?
![Page 97: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/97.jpg)
Sensitivity to affine change
![Page 98: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/98.jpg)
Feature matching
• for a feature x, he found the closest feature x1and the second closest feature x2. If the distance ratio of d(x, x1) and d(x, x2) is smaller than 0.8, then it is accepted as a match.
![Page 99: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/99.jpg)
SIFT flow
![Page 100: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/100.jpg)
Maxima in D
![Page 101: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/101.jpg)
Remove low contrast
![Page 102: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/102.jpg)
Remove edges
![Page 103: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/103.jpg)
SIFT descriptor
![Page 104: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/104.jpg)
![Page 105: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/105.jpg)
Estimated rotation
• Computed affine transformation from rotated image to original image:0.7060 -0.7052 128.42300.7057 0.7100 -128.9491
0 0 1.0000
• Actual transformation from rotated image to original image:0.7071 -0.7071 128.69340.7071 0.7071 -128.6934
0 0 1.0000
![Page 106: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/106.jpg)
SIFT extensions
![Page 107: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/107.jpg)
PCA
![Page 108: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/108.jpg)
PCA-SIFT
• Only change step 4• Pre-compute an eigen-space for local gradient
patches of size 41x41• 2x39x39=3042 elements• Only keep 20 components• A more compact descriptor
![Page 109: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/109.jpg)
GLOH (Gradient location-orientation histogram)
17 location bins16 orientation binsAnalyze the 17x16=272-d eigen-space, keep 128 components
SIFT is still considered the best.
SIFT
![Page 110: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/110.jpg)
Multi-Scale Oriented Patches• Simpler than SIFT. Designed for image matching.
[Brown, Szeliski, Winder, CVPR’2005]• Feature detector
– Multi-scale Harris corners– Orientation from blurred gradient– Geometrically invariant to rotation
• Feature descriptor– Bias/gain normalized sampling of local patch (8x8)– Photometrically invariant to affine changes in
intensity
![Page 111: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/111.jpg)
Multi-Scale Harris corner detector
• Image stitching is mostly concerned with matching images that have the same scale, so sub-octave pyramid might not be necessary.
2s
![Page 112: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/112.jpg)
Multi-Scale Harris corner detector
smoother version of gradients
Corner detection function:
Pick local maxima of 3x3 and larger than 10
![Page 113: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/113.jpg)
Keypoint detection function
Experiments show roughly the same performance.
![Page 114: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/114.jpg)
Non-maximal suppression
• Restrict the maximal number of interest points, but also want them spatially well distributed
• Only retain maximums in a neighborhood of radius r.
• Sort them by strength, decreasing r from infinity until the number of keypoints (500) is satisfied.
![Page 115: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/115.jpg)
Non-maximal suppression
![Page 116: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/116.jpg)
Sub-pixel refinement
![Page 117: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/117.jpg)
Orientation assignment
• Orientation = blurred gradient
![Page 118: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/118.jpg)
Descriptor Vector• Rotation Invariant Frame
– Scale-space position (x, y, s) + orientation ()
![Page 119: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/119.jpg)
MSOP descriptor vector• 8x8 oriented patch sampled at 5 x scale. See TR
for details. • Sampled from with
spacing=5
8 pixels
![Page 120: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/120.jpg)
MSOP descriptor vector• 8x8 oriented patch sampled at 5 x scale. See TR
for details. • Bias/gain normalisation: I’ = (I – )/• Wavelet transform
8 pixels
![Page 121: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/121.jpg)
Detections at multiple scales
![Page 122: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/122.jpg)
Summary
• Multi-scale Harris corner detector• Sub-pixel refinement• Orientation assignment by gradients• Blurred intensity patch as descriptor
![Page 123: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/123.jpg)
Feature matching• Exhaustive search
– for each feature in one image, look at all the other features in the other image(s)
• Hashing– compute a short descriptor from each feature vector,
or hash longer descriptors (randomly)
• Nearest neighbor techniques– k-trees and their variants (Best Bin First)
![Page 124: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/124.jpg)
Wavelet-based hashing• Compute a short (3-vector) descriptor from an
8x8 patch using a Haar “wavelet”
• Quantize each value into 10 (overlapping) bins (103 total entries)
• [Brown, Szeliski, Winder, CVPR’2005]
![Page 125: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/125.jpg)
Nearest neighbor techniques• k-D tree
and
• Best BinFirst(BBF)
Indexing Without Invariants in 3D Object Recognition, Beis and Lowe, PAMI’99
![Page 126: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/126.jpg)
Applications
![Page 127: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/127.jpg)
Recognition
SIFT Features
![Page 128: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/128.jpg)
3D object recognition
![Page 129: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/129.jpg)
3D object recognition
![Page 130: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/130.jpg)
Office of the past
Video of desk Images from PDF
Track & recognize
T T+1
Internal representation
Scene Graph
Desk Desk
![Page 131: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/131.jpg)
…> 5000images
change in viewing angle
Image retrieval
![Page 132: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/132.jpg)
22 correct matches
Image retrieval
![Page 133: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/133.jpg)
…> 5000images
change in viewing angle+ scale change
Image retrieval
![Page 134: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/134.jpg)
Robot location
![Page 135: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/135.jpg)
Robotics: Sony Aibo
SIFT is used for Recognizing
charging station Communicating
with visual cards Teaching object
recognition
soccer
![Page 136: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/136.jpg)
Structure from Motion
• The SFM Problem– Reconstruct scene geometry and camera motion
from two or more images
Track2D Features Estimate
3D Optimize(Bundle Adjust) Fit Surfaces
SFM Pipeline
![Page 137: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/137.jpg)
Structure from Motion
Poor mesh Good mesh
![Page 138: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/138.jpg)
Augmented reality
![Page 139: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/139.jpg)
Automatic image stitching
![Page 140: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/140.jpg)
Automatic image stitching
![Page 141: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/141.jpg)
Automatic image stitching
![Page 142: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/142.jpg)
Automatic image stitching
![Page 143: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/143.jpg)
Automatic image stitching
![Page 144: Features - 國立臺灣大學cyy/courses/vfx/19... · Harris corner detector Measure of corner response: (k – empirical constant, k = 0.04-0.06) 2 ( ) 4 10 01 2 a00 a11 a00 a11](https://reader033.vdocument.in/reader033/viewer/2022043006/5f8f8fc5f72d12104d2627bc/html5/thumbnails/144.jpg)
Reference• Chris Harris, Mike Stephens, A Combined Corner and Edge Detector,
4th Alvey Vision Conference, 1988, pp147-151. • David G. Lowe, Distinctive Image Features from Scale-Invariant
Keypoints, International Journal of Computer Vision, 60(2), 2004, pp91-110.
• Yan Ke, Rahul Sukthankar, PCA-SIFT: A More Distinctive Representation for Local Image Descriptors, CVPR 2004.
• Krystian Mikolajczyk, Cordelia Schmid, A performance evaluation of local descriptors, Submitted to PAMI, 2004.
• SIFT Keypoint Detector, David Lowe.• Matlab SIFT Tutorial, University of Toronto.