image alignment by image averaging david hong ncssm, ie364 2008
TRANSCRIPT
![Page 1: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/1.jpg)
Image Alignment byImage Averaging
David Hong
NCSSM, IE364 2008
![Page 2: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/2.jpg)
Example Problem 1
In:
Out: 4 0 0s s sX Y
![Page 3: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/3.jpg)
Example Problem 2
In:
Out: 0 4 0s s sX Y
![Page 4: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/4.jpg)
Example Problem 3
In:
Out: 0 0 6s s sX Y
![Page 5: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/5.jpg)
Problem (Formal Statement)
In: , n nu v R
Out: , , such that
translated by and and
rotated by is
s s s
s s
s
X Y
v X Y
u
![Page 6: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/6.jpg)
Motivation
Many Applications:– Special Effects (Movie)– Video Compression– Pattern Recognition– Image Stabilization (Digital Cameras)– Dead-reckoning (Mobile Robotics)
![Page 7: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/7.jpg)
State of the Art
Optical Flow
Lucas-Kanade (1985)
Optical Flow with Smoothness Constraint
Horn-Schunck (1980)
Phase Correlation
![Page 8: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/8.jpg)
X
Y
u(x,y,t)
U(X,Y)
xy
Lucas-Kanade
![Page 9: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/9.jpg)
Lucas-Kanade
0 0, , ,U X Y u x y t
Then the floor-coordinate is (X0,Y0) and the sensor-coordinate is (x,y) at time t.
Let us consider a point on the plane.
From there, we can see:
![Page 10: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/10.jpg)
Lucas-Kanade
0 0, , ,
0
U X Y u x y t
d dU u
dt dtdx
u u udtdyx y t
dt
Differentiating on time gives us:
![Page 11: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/11.jpg)
Lucas-Kanade
0
0
cos sin
sin coss s s
s s s
X Xx
Y Yy
Expressing (x,y) in terms of (X0,Y0) and the sensor position (Xs,Ys,Θs) gives us:
![Page 12: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/12.jpg)
Lucas-Kanade
s
s
s
dX
dtdYu u u u u
x yx y y x dt t
d
dt
Putting the two together, we get:
This is underdetermined!
![Page 13: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/13.jpg)
Algorithm
1, 1, , 1 , 1 , 1 , 1 1, 1, , ,2 's
i j i j i j i j i j i j i j i j s i j i j
s
X
u u u u u u i u u j Y u u
uu’
![Page 14: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/14.jpg)
Improvement by Iteration
u
u’
Iteration 1:
3
2
0
s
s
s
X
Y
u’’
Iteration 2:
1
1
0
s
s
s
X
Y
Overall Estimate:
4
1
0
s
s
s
X
Y
u’’’
![Page 15: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/15.jpg)
Improvement by Iteration
u’
u
u’’
u’(x’,y’)
u’’(x’’,y’’)
![Page 16: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/16.jpg)
Improvement by Iteration
Coordinate Transformation:
cos sin' ''
sin cos' ''ss s
s s s
Xx x
y y Y
![Page 17: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/17.jpg)
Improvement by Iteration
u’
Places u’ is defined
Place we need to evaluate u’
(x’, y’)
(x’0, y’0) (x’1, y’0)
(x’0, y’1) (x’1, y’1)
![Page 18: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/18.jpg)
Improvement by Iteration
1 10 0
0 1 0 1
1 00 1
0 1 1 0
0 11 0
1 0 0 1
0 0
1
Lagrange Interpolation:
' ' ' ''( ', ') ' ' , '
' ' ' '
' ' ' ' ' ' , '
' ' ' '
' ' ' ' ' ' , '
' ' ' '
' ' ' '
'
x x y yu x y u x y
x x y y
x x y yu x y
x x y y
x x y yu x y
x x y y
x x y y
x x
1 10 1 0
' ' , '' ' '
u x yy y
![Page 19: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/19.jpg)
Improvement by Iteration
u’
u
u’’
u’ not defined!
![Page 20: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/20.jpg)
Improvement by Iteration
u’
u’’
u’’ was not evaluated here
Valid Region
![Page 21: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/21.jpg)
Improvement by Iteration
Places we need to evaluate u’
(i-1,0’, j-1,0’ )
(i0,-1’, j0,-1’ )
(i0,1’, j0,1’ )
(i1,0’, j1,0’ )
![Page 22: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/22.jpg)
Performance of Algorithm
Good Surface: Bad Surface:
Algorithm Fails!
![Page 23: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/23.jpg)
Performance of Algorithm
Surface:
![Page 24: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/24.jpg)
Performance of AlgorithmSurface:
![Page 25: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/25.jpg)
Performance of AlgorithmSurface:
![Page 26: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/26.jpg)
Assumptions Made
• The Error Function is locally quadratic
• The floor is linear
![Page 27: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/27.jpg)
Weaknesses
• Many Iterations– Inherent to Technique
• “Fooled” by symmetry (Aliasing problem)– Inherent to Problem
![Page 28: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/28.jpg)
Strengths
• Accurate
• Improvement by Iteration
• Finds Error Function Root by Newton’s
![Page 29: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/29.jpg)
yu
v(xv,yv)
u(xu,yu)
x v
y v
Phase Correlation
xu
![Page 30: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/30.jpg)
Phase Correlation
We consider the image to be like a 2-D wave.
Then, displacement is simply a “phase shift”
Rotation can similarly be found
So, we “correlate” the “phases”
![Page 31: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/31.jpg)
Phase Correlation
Discrete Fourier Transform:
1
1
( )
ijij
ijij
ij ij
Fn
Gn
FG
![Page 32: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/32.jpg)
Phase CorrelationKey Theorem:
,where is the motion by displacement and
is the desired output.
d d
d
d
F W F
d
W
' d dFz F z W Fz
![Page 33: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/33.jpg)
Phase Correlation
1
2
Key Theorem:
' : such that '
:
( ')( ) ( )( )
m
i vd
Let z z R x d z x
Then
F z v e F z Rv
R R
![Page 34: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/34.jpg)
Phase Correlation
Corollary:
( ) ( )
'( ) ' ( )
:
'
Let u v F z v
Let u v F z v
Then
u v u Rv
![Page 35: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/35.jpg)
Weaknesses
• Inaccurate on first iteration
• Boundary Problem (Repetion Assumption)
• High complexity– FFT is “O(nlogn)”
![Page 36: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/36.jpg)
Strengths
• Elegant
• Makes a big leap
• Works well on images with pattern
• Separates displacement and rotation (DFT)
![Page 37: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/37.jpg)
yu
v(xv,yv)
u(xu,yu)
x v
y v
Image Averaging
xu
![Page 38: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/38.jpg)
Image Averaging
• Find a “Center-of-Mass” of each image
• Track the motion of the center-of-mass
w x xx
w x
![Page 39: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/39.jpg)
Weaknesses
• Boundary Problem (Average Point Moves)
• Average is affected by small discretization issues
![Page 40: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/40.jpg)
Strengths
• Elegant• Makes a big leap• Very fast– Complexity of O(n)
• Yields itself well to Improvement by Iteration– Using same technique as in Lucas-Kanade
![Page 41: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/41.jpg)
Handling the Weaknesses
• Here we decide to take an alternative approach
• Separate displacements from rotation
• Do this using FFT (as in Phase Correlation)
![Page 42: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/42.jpg)
Handling the Weaknesses
• We handle rotation first– Post-FFT, only rotation remains
![Page 43: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/43.jpg)
Handling the Weaknesses
![Page 44: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/44.jpg)
Weaknesses
• We introduce an FFT ( O(nlogn) operation)
• However, only requires 2– Phase correlation requires up to 3 or 4
![Page 45: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/45.jpg)
Future Work
• Make Image Alignment Rigorous– Use complex numbers to notate displacement
• Smoothness Constraint• Pre-processing the image• Condition for Convergence
![Page 46: Image Alignment by Image Averaging David Hong NCSSM, IE364 2008](https://reader030.vdocument.in/reader030/viewer/2022032606/56649e975503460f94b9aa2b/html5/thumbnails/46.jpg)
Thank You!