![Page 1: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/1.jpg)
Optical Flow Methods
CISC 489/689Spring 2009
University of Delaware
![Page 2: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/2.jpg)
Outline
• Review of Optical Flow Constraint, Lucas-Kanade, Horn and Schunck Methods
• Lucas-Kanade Meets Horn and Schunck• 3D Regularization• Techniques for solving optical flow• Confidence Measures in Optical Flow
![Page 3: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/3.jpg)
Optical Flow Constraint
0
0
),,(),,(
),,(),,(
tyx fvfuf
t
f
dt
dy
y
f
dt
dx
x
f
tyxft
ft
y
fy
x
fxtyxf
tyxftttvytuxf
0limit takingand by Dividing tt
),( tvytux
),( yx
),,( tyxf
),,( ttyxf
![Page 4: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/4.jpg)
Interpretation
Constraint Line
tT
tyx
fv
uf
fvfuf
0
yx ff ,
v
u 0
1
v
u
fff tyx
![Page 5: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/5.jpg)
Lucas-Kanade Method
* = JK J
1
*),( v
u
JKvuELK
2
2
2
2
2
2
0
1
ttytx
tyyyx
txyxx
ttytx
tyyyx
txyxx
fffff
fffff
fffff
J
v
u
fffff
fffff
fffff
![Page 6: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/6.jpg)
Lucas-Kanade Method
• Local Method, window based• Cannot solve for optical flow everywhere• Robust to noise
5.7 15
Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005
![Page 7: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/7.jpg)
Dense optical Flow
?)(),( Minimize 2 tyx fvfufvuE
Lacks Smoothness
Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005
![Page 8: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/8.jpg)
Horn and Schunck MethoddxdyvufvfufvuE tyxHS )()(),(
222
Euler-Lagrange Equations
![Page 9: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/9.jpg)
Horn and Schunck Method
510 610Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005
• Global Method • Estimates flow everywhere• Sensitive to noise• Oversmooths the edges
![Page 10: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/10.jpg)
Why combine them?
• Need dense flow estimate• Robust to noise• Preserve discontinuities
![Page 11: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/11.jpg)
Combining the two…
1
*),( v
u
JKvuELK
dxdyvufvfufvuE tyxHS )()(),(222
dxdyvuv
u
JKvuECLG )(
1
*),(22
![Page 12: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/12.jpg)
Combined Local Global Method
dxdyvuv
u
JKvuECLG )(
1
*),(22
Euler-Lagrange Equations
AverageError
Standard Deviation
Lucas&Kanade 4.3(density 35%)
Horn&Schunk 9.8 16.2
Combining local and global 4.2 7.7
Table: Courtesy - Darya Frolova, Recent progress in optical flow
![Page 13: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/13.jpg)
Preserving discontinuities• Gaussian Window does not preserve
discontinuities• Solutions
– Use bilateral filtering
– Add gradient constancy
dxdyvuv
u
JKvuE bilbil )(
1
*),(22
dxdyffvuv
u
JKvuE tgrad2
1
22)()(
1
*),(
![Page 14: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/14.jpg)
Bilateral support window
Images: Courtesy, Darya Frolova, Recent progress in optical flow
![Page 15: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/15.jpg)
Robust statistics – simple exampleFind “best” representative for the set of numbers
min2
iixxEL2: min
iixxEL1:
xix
Influence of xi on E: xi → xi + ∆
)mean( ixx
Outliers influence the most
ixx proportional to
xxEE ioldnew 2
)median( ixx
Majority decides
equal for all xi
oldnew EE
Slide: Courtesy - Darya Frolova, Recent progress in optical flow
![Page 16: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/16.jpg)
Robust statistics
many ordinary people a very rich man
Oligarchy
Votes proportional to the wealth
Democracy
One vote per person
wealth
like in L2 norm minimization like in L1 norm minimization
Slide: Courtesy - Darya Frolova, Recent progress in optical flow
![Page 17: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/17.jpg)
Combination of two flow constraints
robust: L1
x
robust in presence of outliers– non-smooth: hard to analyze
usual: L2
easy to analyze and minimize– sensitive to outliers
2x
robust regularized
smooth: easy to analyze robust in presence of outliers
22 x
ε
video
warpedwarped IIII min
),,( ; )1,,( tyxIItvyuxIIwarped
[A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performanceSlide: Courtesy - Darya Frolova, Recent progress in optical flow
![Page 18: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/18.jpg)
Robust statistics
![Page 19: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/19.jpg)
3D Regularization
• accounted for spatial regularization
• If velocities do not change suddenly with time, can we regularize in time as well?
)(22vu
![Page 20: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/20.jpg)
3D Regularization
t
v
y
v
x
vv
t
u
y
u
x
uu
3
3
dxdydtvuv
u
JKvuETXCLG )(
1
*),(2
3
2
3],0[3
![Page 21: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/21.jpg)
Multiresolution estimation
21
image Iimage J
Gaussian pyramid of image 1 Gaussian pyramid of image 2
Image 2image 1
run iterative estimation
run iterative estimation
warp & upsample
.
.
.
![Page 22: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/22.jpg)
Multi-resolution Lucas Kanade Algorithm
Compute Iterative LK at highest level•For Each Level i• Take flow u(i-1), v(i-1) from level i-1
• Upsample the flow to create u*(i), v*(i) matrices of twice resolution for level i.
• Multiply u*(i), v*(i) by 2
• Compute It from a block displaced by u*(i), v*(i)
• Apply LK to get u’(i), v’(i) (the correction in flow)
• Add corrections u’(i), v’(i) to obtain the flow u(i), v(i) at the ith level, i.e., u(i)=u*(i)+u’(i), v(i)=v*(i)+v’(i)
![Page 23: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/23.jpg)
Comparison of errors
For Yosemite sequence with cloudsTable: Courtesy - Darya Frolova, Recent progress in optical flow
![Page 24: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/24.jpg)
Solving the system
fAu
How to solve?
2 components of success: fast convergence
good initial guess
Start with some initial guess
and apply some iterative method
initialu
![Page 25: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/25.jpg)
Relaxation schemes have smoothing property:
Only neighboring pixels are coupled in relaxation
scheme
It may take thousands of iterations to propagate
information to large distance
. . . . . . . . . . . .
Relaxation smoothes the error
![Page 26: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/26.jpg)
Relaxation smoothes the error Examples
2D case:
1D case:
Error of initial guess Error after 5 relaxation Error after 15 relaxations
![Page 27: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/27.jpg)
Idea: coarser grid
On a coarser grid low frequencies become higher
Hence, relaxations can be more effective
initial grid – fine grid
coarse grid – we take every second point
![Page 28: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/28.jpg)
Multigrid 2-Level V-Cycle
1. Iterate ⇒ error becomes smooth
2. Transfer error equation to the coarse level ⇒ low frequencies become high
3. Solve for the error on the coarse level ⇒ good error estimation
4. Transfer error to the fine level
5. Correct the previous solution6. Iterate ⇒ remove interpolation artifacts
![Page 29: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/29.jpg)
make iteration process faster (on the coarse grid we can effectively minimize the error)
obtain better initial guess (solve directly on the coarsest grid)
Coarse grid - advantages
initialu
fAu
Coarsening allows:
go to the coarsest grid
solve here the equation
to findinitialu
interpolate to the
finer grid
initialu
![Page 30: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/30.jpg)
Multigrid approach – Full scheme
![Page 31: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/31.jpg)
Confidence Metric
• Intrinsic in Local Methods• How to evaluate for global methods?
– Edge strength?• Doesn’t work (Barron et al.,1994)
![Page 32: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/32.jpg)
Confidence Metric
• Histogram of error contribution
Error
Number of pixels
![Page 33: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/33.jpg)
Confidence Metric
![Page 34: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/34.jpg)
More Results
![Page 35: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/35.jpg)
More Results
![Page 36: Optical Flow Methods CISC 489/689 Spring 2009 University of Delaware](https://reader038.vdocument.in/reader038/viewer/2022110205/56649c745503460f949272ba/html5/thumbnails/36.jpg)
Further Reading• Combining the advantages of local and global optic flow methods
(“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005
• High accuracy optical flow estimation based on a theory for warping
T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005• Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 -
2005• Towards ultimate motion estimation: Combining highest accuracy
with real-time performance. A. Bruhn, J. Weickert, 2005• Bilateral filtering-based optical flow estimation with occlusion
detection. J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006