14.4 alignment - lucaskanade16385/s17/slides/14.4... · idea #1: template matching slow,...
TRANSCRIPT
![Page 1: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/1.jpg)
Image Alignment16-385 Computer Vision (Kris Kitani)
Carnegie Mellon University
Lucas Kanade
![Page 2: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/2.jpg)
http://www.humansensing.cs.cmu.edu/intraface/
![Page 3: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/3.jpg)
![Page 4: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/4.jpg)
How can I find in the image?
![Page 5: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/5.jpg)
Idea #1: Template Matching
Slow, combinatory, global solution
![Page 6: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/6.jpg)
Idea #2: Pyramid Template Matching
Faster, combinatory, locally optimal
![Page 7: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/7.jpg)
Idea #3: Model refinement
Fastest, locally optimal
(when you have a good initial solution)
![Page 8: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/8.jpg)
Some notation before we get into the math…
W(x;p)2D image transformation
x =
x
y
�
p = {p1, . . . , pN}
2D image coordinate
Parameters of the transformation
Translation
W(x;p) =
x+ p1
y + p2
�
=
1 0 p1
0 1 p2
�2
4x
y
1
3
5
transform coordinate
Affine
Pixel value at a coordinate
I(x0) = I(W(x;p))
W(x;p) =
p1x+ p2y + p3
p4x+ p5y + p6
�
=
p1 p2 p3
p4 p5 p6
�2
4x
y
1
3
5
affine transformcoordinate
Warped image
can be written in matrix form when linear affine warp matrix can also be 3x3 when last row is [0 0 1]
![Page 9: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/9.jpg)
W(x;p)
p = {p1, . . . , pN}
is a function of ____ variables
W(x;p) takes a ________ as input and returns a _______
W(x;p) returns a ______ of dimension ___ x ___
where N is _____ for an affine model
I(x0) = I(W(x;p)) this warp changes pixel values?
![Page 10: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/10.jpg)
Image alignment (problem definition)
minp
X
x
[I(W(x;p))� T (x)]2
warped image template image
Find the warp parameters p such that the SSD is minimized
![Page 11: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/11.jpg)
T (x)
W(x;p)
I(x)
Find the warp parameters p such that the SSD is minimized
![Page 12: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/12.jpg)
Image alignment (problem definition)
minp
X
x
[I(W(x;p))� T (x)]2
warped image template image
Find the warp parameters p such that the SSD is minimized
How could you find a solution to this problem?
![Page 13: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/13.jpg)
minp
X
x
[I(W(x;p))� T (x)]2
This is a non-linear (quadratic) function of a non-parametric function!
(Function I is non-parametric)
Hard to optimize
What can you do to make it easier to solve?
![Page 14: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/14.jpg)
minp
X
x
[I(W(x;p))� T (x)]2
Hard to optimize
What can you do to make it easier to solve?
assume good initialization, linearized objective and update incrementally
This is a non-linear (quadratic) function of a non-parametric function!
(Function I is non-parametric)
![Page 15: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/15.jpg)
If you have a good initial guess p…X
x
[I(W(x;p))� T (x)]2
can be written as …X
x
[I(W(x;p+�p))� T (x)]2
(a small incremental adjustment)
(pretty strong assumption)
(this is what we are solving for now)
![Page 16: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/16.jpg)
X
x
[I(W(x;p+�p))� T (x)]2
How can we linearize the function I for a really small perturbation of p?
Taylor series approximation!Hint:
This is still a non-linear (quadratic) function of a non-parametric function!
(Function I is non-parametric)
![Page 17: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/17.jpg)
X
x
[I(W(x;p+�p))� T (x)]2
How can we linearize the function I for a really small perturbation of p?
Taylor series approximation!
This is still a non-linear (quadratic) function of a non-parametric function!
(Function I is non-parametric)
![Page 18: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/18.jpg)
X
x
[I(W(x;p+�p))� T (x)]2
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2Linear approximation
Multivariable Taylor Series Expansion (First order approximation)
f(x, y) ⇡ f(a, b) + f
x
(a, b)(x� a)� f
y
(a, b)(y � b)
Is this a linear function of the unknowns?
![Page 19: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/19.jpg)
Multivariable Taylor Series Expansion (First order approximation)
f(x, y) ⇡ f(a, b) + f
x
(a, b)(x� a)� f
y
(a, b)(y � b)
x
0 = W(x;p)Recall:
I(W(x;p+�p)) ⇡ I(W(x;p)) +@I(W(x;p)
@p�p
= I(W(x;p)) +@I(W(x;p)
@x0@W(x;p)
@p�p
= I(W(x;p)) +rI@W
@p�p
chain rule
short-hand
short-hand
![Page 20: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/20.jpg)
X
x
[I(W(x;p+�p))� T (x)]2
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2Linear approximation
Multivariable Taylor Series Expansion (First order approximation)
f(x, y) ⇡ f(a, b) + f
x
(a, b)(x� a)� f
y
(a, b)(y � b)
Now, the function is a linear function of the unknowns
![Page 21: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/21.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
is a function of _____ variables
is a _________ of dimension ___ x ___
is a _________ of dimension ___ x ___
is a __________ of dimension ___ x ___
Wp
I(·)
x
output of
![Page 22: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/22.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
is a _________ of dimension ___ x ___rI
@W
@pis a _________ of dimension ___ x ___
�p is a _________ of dimension ___ x ___
(I haven’t explained this yet)
![Page 23: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/23.jpg)
The Jacobian @W
@p
@W
@p=
2
64
@Wx
@p1
@Wx
@p2· · · @W
x
@pN
@Wy
@p1
@Wy
@p2· · · @W
y
@pN
3
75
Affine transform
@W
@p=
x 0 y 0 1 00 x 0 y 0 1
�
@W
x
@p1= x
@Wy
@p1= 0
@Wx
@p2= 0 · · ·
· · ·
W(x;p) =
p1x+ p3y + p5
p2x+ p4y + p6
�
Rate of change of the warp
(A matrix of partial derivatives)
x =
x
y
�
W =
W
x
(x, y)W
y
(x, y)
�
![Page 24: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/24.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
![Page 25: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/25.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
pixel coordinate (2 x 1)
![Page 26: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/26.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
pixel coordinate (2 x 1)
image intensity (scalar)
![Page 27: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/27.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
pixel coordinate (2 x 1)
image intensity (scalar)
warp function (2 x 1)
![Page 28: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/28.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
pixel coordinate (2 x 1)
image intensity (scalar)
warp function (2 x 1)
warp parameters (6 for affine)
![Page 29: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/29.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
pixel coordinate (2 x 1)
image intensity (scalar)
warp function (2 x 1)
warp parameters (6 for affine)
image gradient (1 x 2)
![Page 30: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/30.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
pixel coordinate (2 x 1)
image intensity (scalar)
warp function (2 x 1)
warp parameters (6 for affine)
image gradient (1 x 2)
Partial derivatives of warp function (2 x 6)
![Page 31: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/31.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
pixel coordinate (2 x 1)
image intensity (scalar)
warp function (2 x 1)
warp parameters (6 for affine)
image gradient (1 x 2)
Partial derivatives of warp function (2 x 6)
incremental warp (6 x 1)
![Page 32: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/32.jpg)
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
pixel coordinate (2 x 1)
image intensity (scalar)
warp function (2 x 1)
warp parameters (6 for affine)
image gradient (1 x 2)
Partial derivatives of warp function (2 x 6)
incremental warp (6 x 1)
template image intensity (scalar)
When you implement this, you will compute everything in parallel and store as matrix … don’t loop over x!
![Page 33: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/33.jpg)
Summary (of Lucas-Kanade Image Alignment)
Solve for increment
minp
X
x
[I(W(x;p))� T (x)]2
X
x
[I(W(x;p+�p))� T (x)]2
Taylor series approximation Linearize
Difficult non-linear optimization problem
Assume known approximate solution
Strategy:
Problem:
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
then solve for �p
warped image template image
![Page 34: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/34.jpg)
OK, so how do we solve this?
min�p
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
![Page 35: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/35.jpg)
min�p
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
min�p
X
x
rI
@W
@p�p� {T (x)� I(W(x;p))}
�2
Another way to look at it…
(moving terms around)
Have you seen this form of optimization problem before?
vector of variables
vector of constants constant
![Page 36: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/36.jpg)
min�p
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
min�p
X
x
rI
@W
@p�p� {T (x)� I(W(x;p))}
�2
Ax � b
Another way to look at it…
How do you solve this?
Looks like
variableconstant constant
![Page 37: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/37.jpg)
Least squares approximation
x̂ = argminx
||Ax� b||2 is solved byx = (A>
A)�1A
>b
min�p
X
x
rI
@W
@p�p� {T (x)� I(W(x;p))}
�2
is optimized when
where A>A
x = (A>A)�1
A
>b
H =X
x
rI
@W
@p
�> rI
@W
@p
�
�p = H�1X
x
rI
@W
@p
�>[T (x)� I(W(x;p))]
Applied to our tasks:
after applying
![Page 38: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/38.jpg)
Solve for increment
minp
X
x
[I(W(x;p))� T (x)]2
X
x
[I(W(x;p+�p))� T (x)]2
Taylor series approximation Linearize
Difficult non-linear optimization problem
Assume known approximate solution
Strategy:
Solve:
X
x
I(W(x;p)) +rI
@W
@p�p� T (x)
�2
warped image template image
Solution:Solution to least squares
approximation
HessianH =X
x
rI
@W
@p
�> rI
@W
@p
�
�p = H�1X
x
rI
@W
@p
�>[T (x)� I(W(x;p))]
![Page 39: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/39.jpg)
Gauss-Newton gradient decent non-linear optimization!
This is called…
![Page 40: 14.4 Alignment - LucasKanade16385/s17/Slides/14.4... · Idea #1: Template Matching Slow, combinatory, global solution. Idea #2: Pyramid Template Matching Faster, combinatory, locally](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed1e5aa76055c3966516686/html5/thumbnails/40.jpg)
1. Warp image
2. Compute error image
3. Compute gradient
4. Evaluate Jacobian
5. Compute Hessian
6. Compute
7. Update parameters
[T (x)� I(W(x;p))]2
I(W(x;p))
@W
@p
H
�p
p p+�p
Lucas Kanade (Additive alignment)
�p = H�1X
x
rI
@W
@p
�>[T (x)� I(W(x;p))]
H =X
x
rI
@W
@p
�> rI
@W
@p
�
Just 8 lines of code!
rI(x0) x’coordinates of the warped image (gradients of the warped image)