optical flow: horn-schunck16385/s17/slides/14.3_of__hornschunck.pdf · optical flow: horn-schunck...
Post on 07-Jun-2020
9 Views
Preview:
TRANSCRIPT
Optical Flow: Horn-Schunck16-385 Computer Vision (Kris Kitani)
Carnegie Mellon University
Horn-Schunck Optical Flow (1981)
Lucas-KanadeOptical Flow (1981)
‘constant’ flow(flow is constant for all pixels)
‘smooth’ flow(flow can vary from pixel to pixel)
brightness constancymethod of differences
global method (dense)
local method (sparse)
small motion
Optical Flow (Problem definition)
Estimate the motion (flow) between these
two consecutive images
I(x, y, t)I(x, y, t0)
How is this different from estimating a 2D transform?
Key Assumptions (unique to optical flow)
Color Constancy(Brightness constancy for intensity images)
Small Motion(pixels only move a little bit)
Implication: allows for pixel to pixel comparison (not image features)
Implication: linearization of the brightness constancy constraint
Approach
I(x, y, t)I(x, y, t0)
Look for nearby pixels with the same color(small motion) (color constancy)
Brightness constancy
I(x, y, t)
(x, y)
(x+ u�t, y + v�t)
(x, y)
(u, v) (�x, �y) = (u�t, v�t)Optical flow (velocities): Displacement:
I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)
I(x, y, t+ �t)
For a really small time step…
Optical Flow Constraint equationI(x+ u�t, y + v�t, t+ �t) = I(x, y, t)
I(x, y, t) +@I
@x
�x+@I
@y
�y +@I
@t
�t = I(x, y, t)
Ix
u+ Iy
v + It
= 0
@I
@x
dx
dt
+@I
@y
dy
dt
+@I
@t
= 0
Ix
u+ Iy
v + It
= 0
u
v
Solution lies on a straight line
The solution cannot be determined uniquely with a single constraint (a single pixel)
Where can we get an additional constraint?
Horn-Schunck Optical Flow (1981)
Lucas-KanadeOptical Flow (1981)
‘constant’ flow(flow is constant for all pixels)
‘smooth’ flow(flow is smooth from pixel to pixel)
brightness constancymethod of differences
global method local method
small motion
Smoothness
most objects in the world are rigid or deform elastically
moving together coherently
we expect optical flow fields to be smooth
Key idea (of Horn-Schunck optical flow)
Enforce brightness constancy
Enforce smooth flow field
to compute optical flow
Key idea (of Horn-Schunck optical flow)
Enforce brightness constancy
Enforce smooth flow field
to compute optical flow
Enforce brightness constancy
Ix
u+ Iy
v + It
= 0
minu,v
Ix
uij
+ Iy
vij
+ It
�2For every pixel,
Ix
u+ Iy
v + It
= 0
minu,v
Ix
uij
+ Iy
vij
+ It
�2For every pixel,
lazy notation for Ix
(i, j)
Enforce brightness constancy
Ix
Iy
changes these
displacement of object
u v changes these
It
+ =
Find the optical flow such that it satisfies:
Key idea (of Horn-Schunck optical flow)
Enforce brightness constancy
Enforce smooth flow field
to compute optical flow
Enforce smooth flow field
uij ui+1,jui�1,j
ui,j+1
ui,j�1
minu
(ui,j � ui+1,j)2
u-component of flow
Which flow field optimizes the objective?
X
ij
(uij � ui+1,j)2
X
ij
(uij � ui+1,j)2?
minu
(ui,j � ui+1,j)2
smallbig
Which flow field optimizes the objective? minu
(ui,j � ui+1,j)2
Key idea (of Horn-Schunck optical flow)
Enforce brightness constancy
Enforce smooth flow field
to compute optical flow
bringing it all together…
Horn-Schunck optical flow
minu,v
X
i,j
⇢Es(i, j) + �Ed(i, j)
�smoothness brightness constancy
weight
HS optical flow objective function
Brightness constancy
Smoothness
(uij � ui+1,j)
ij
i, j + 1
i+ 1, j ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
i� 1, j
i, j � 1
(uij � ui,j+1)
ij
i, j + 1
i+ 1, j ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
i� 1, j
i, j � 1
(vij � vi+1,j) (vij � vi,j+1)
Ed
(i, j) =
Ix
uij
+ Iy
vij
+ It
�2
Es(i, j) =1
4
(uij � ui+1,j)
2 + (uij � ui,j+1)2 + (vij � vi+1,j)
2 + (vij � vi,j+1)2
�
why not all four neighbors?
How do we solve this minimization problem?
minu,v
X
i,j
⇢Es(i, j) + �Ed(i, j)
�
Compute partial derivative, derive update equations (gradient decent!)
minu,v
X
i,j
⇢Es(i, j) + �Ed(i, j)
�
How do we solve this minimization problem?
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
smoothness term brightness constancy
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
it’s not so bad…
how many u terms depend on k and l?
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
it’s not so bad…
how many u terms depend on k and l?ONE from brightness constancyFOUR from smoothness
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
it’s not so bad…
how many u terms depend on k and l?ONE from brightness constancyFOUR from smoothness
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
(uij � ui+1,j)
ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
(u2ij � 2uijui+1,j + u2
i+1,j) (u2ij � 2uijui,j+1 + u2
i,j+1)
(variable will appear four times in sum)
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
(uij � ui+1,j)
ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
uij =1
4
⇢ui+1,j + ui�1,j + ui,j+1 + ui,j�1
�short hand for local average
(u2ij � 2uijui+1,j + u2
i+1,j) (u2ij � 2uijui,j+1 + u2
i,j+1)
(variable will appear four times in sum)
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
Ax = b
how do you solve this?
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
this is a linear system
ok, take a step back, why are we doing all this math?
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
We are solving for the optical flow (u,v) given two constraints
smoothness brightness constancy
We need the math to minimize this (back to the math)
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
Ax = b
how do you solve this?
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
Partial derivatives of Horn-Schunck objective function E:
x = A�1b =
adjA
detAb
Recall
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
x = A�1b =
adjA
detAb
Recall
(det A)
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
Same as the linear system:{1 + �(I2
x
+ I2y
)}ukl
= (1 + �I2x
)ukl
� �Ix
Iy
vkl
� �Ix
It
{1 + �(I2x
+ I2y
)}vkl
= (1 + �I2y
)vkl
� �Ix
Iy
ukl
� �Iy
It
(det A)
Rearrange to get update equations:
new value
old average
{1 + �(I2x
+ I2y
)}ukl
= (1 + �I2x
)ukl
� �Ix
Iy
vkl
� �Ix
It
{1 + �(I2x
+ I2y
)}vkl
= (1 + �I2y
)vkl
� �Ix
Iy
ukl
� �Iy
It
ukl
= ukl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Ix
vkl
= vkl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Iy
new value
old average
ukl
= ukl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Ix
vkl
= vkl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Iy
When lambda is small (lambda inverse is big)…
minu,v
X
i,j
⇢Es(i, j) + �Ed(i, j)
�Recall:
new value
old average
ukl
= ukl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Ix
vkl
= vkl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Iy
When lambda is small (lambda inverse is big)…goes to
zero
goes to zero
minu,v
X
i,j
⇢Es(i, j) + �Ed(i, j)
�Recall:
new value
old average
ukl
= ukl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Ix
vkl
= vkl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Iy
When lambda is small (lambda inverse is big)…goes to
zero
goes to zero
…we only care about smoothness.
minu,v
X
i,j
⇢Es(i, j) + �Ed(i, j)
�Recall:
ok, take a step back, why did we do all this math?
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
We are solving for the optical flow (u,v) given two constraints
smoothness brightness constancy
We needed the math to minimize this (now to the algorithm)
Horn-Schunck Optical Flow Algorithm
1. Precompute image gradients
2. Precompute temporal gradients
3. Initialize flow field
4. While not converged Compute flow field updates for each pixel:
Ix
Iy
It
u = 0
v = 0
ukl
= ukl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Ix
vkl
= vkl
� Ix
ukl
+ Iy
vkl
+ It
��1 + I2x
+ I2y
Iy
Just 8 lines of code!
top related