professor horst cerjak, 19.12.2005 1 thomas pock a duality based approach for realtime tv-l 1...

Professor Horst Cerjak, 19.12.20051

Thomas Pock A Duality Based Approach for Realtime TV-L1 Optical Flow

ICG

A Duality Based Approach for Realtime TV-L1 Optical Flow

Christopher Zach1, Thomas Pock2, and Horst Bischof2

1 VRVis Research Center, Graz2 Institute for Computer Graphics and Vision, TU Graz

E-mail: {zach, pock, bischof}@icg.tugraz.at



ICG

time

Motivation

• Discontinuity preserving regularization of the flow field

• Robustness to occulsions

• Handle large displacements

• Realtime (> 30 fps) for large images (512x512)



ICG Outline

• (I) Variational Optical Flow

• (II) TV-L1 optical flow

• (III) Duality Based Approach

• (IV) Acceleration using the GPU

• (V) Performance Evaluation

• (VI) Conclusion & Demo



ICG Optical Flow

• Optical Flow (OF) is a major task of biological and artificial visual systems

• Relates the motion of pixel intensities between consecutive image frames

• Optical Flow Constraint:

• Gives only the normal flow• No OFC in untextured areas

0),(),(),( txItxIxu t

u1

u2 u



ICG Variational Optical Flow

• First studied by Horn and Schunck in 1981 [1]

• Quadratic regularization does not allow for discontinuities and occlusions

• Modifying the Horn and Schunck functional was pioneered by Black and Rangarajan [2]

[1] B.K. Horn and B.G. Schunck. Determinig Optical Flow. Artificial Intelligence, 1981[2] M.J. Black and P. Rangarajan. On the Unification of Line Processes, Outlier Rejection and Robust Statistics withApplications in Early Vision, IJCV, 1996

dxxIxuxIdxuuu

201

2

2

2

1 )())((min

)(2 ss



ICG TV-L1 Optical Flow

• We use a robust variant of the Horn-Schunck formulation– Total Variation (TV) of Rudin Osher and Fatemi (ROF) [3]– L1 penalization of the OF constraint

• TV-L1 has been used in many approaches• Allows for discontinuities in the flow field and outliers in

the optical flow constraint• Sophisticated optimization techniques are needed• This is the major goal of this paper

dxxIxuxIdxuuu

)())((min 0121

[3] L. Rudin and S. Osher and E. Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, Physica D, 1992



ICG An Approximative Formulation

dxvvudxuEd

ddd

dvu

2

, 2

1min

dxudxuEd

du

min

Eθ E as Θ 0

Main difficulty is induced by the TV term –> ROF model [3]Simple pointwise optimization problem -> Thresholding

[3] L. Rudin and S. Osher and E. Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, Physica D, 1992



ICG Primal Formulation

• Study of the ROF model

dxvudxuEROF2

2

1min

01

vuu

u

u

• Primal Euler Lagrange equations

• Degenerated if gradient vanishes

2uu• Simple solution: Replace by

• Disadvantage: Large ε smoothes edges!



ICG Dual Formulation

• Studied by Chan [4], and later by Chambolle [5]

0 uup

01

vuu

u

01

vup

0 vpvpp

p• One arrives at two new equations

[4] T. Chan and G. Golub and P. Mulet, A Nonlinear Primal Dual Method for TV-based Image Restoration, 1999[5] A. Chambolle, An Algorithm for Total Variation Minimization and Applications, 2004

• Advantage: No regularization is needed!

• Dual Euler Lagrange Equations

|p| ≤ 1

u



ICG Primal vs. Dual• Convergence of the Primal and Dual formulation

– Primal: fixed-point scheme of Vogel & Oman [6]– Dual: fixed-point scheme of Chambolle [5]

ε=10-15ε=10-1ε=1ε=10EROF

iterations[5] A. Chambolle, An Algorithm for Total Variation Minimization and Applications, 2004[6] C. R. Vogel and M. E. Oman. Iterative Methods For Total Variation Denoising. 1996



ICG Final Algorithm

dxvvudxuEd

ddd

dvu

2

, 2

1min

1.Fix v, minimize wrt. u (Chambolle‘s algorithm)

2.Fix u, minimize wrt. v (Thresholding)

3.Goto 1 until convergence

• Energy minimization is embedded into a coarse-to-fine approach to handle large displacements

• Solved via alternating optimization



ICGImplementation on Graphics

Hardware

• Particularly well suited to compute variational methods– High degree of parallelism– High performance processing units

• All features can be accessed via C-like languages• Performance of graphics cards is steadily increasing

G92

Nov2007



ICG Performance Evaluation

Image resolution 50 Iterations 100 Iterations 200 IterationsGraphics card 7900GTX 8800GTX 7900GTX 8800GTX 7900GTX 8800GTX

128x128 95 125.9 57.6 65.12 30.9 32.82

256x256 34.1 83.11 17.5 43.29 8.9 21.85

512x512 9.3 45.07 4.7 22.93 2.3 11.69

Error evaluation on the well known Yosemite without clouds sequence

HS [1] TV-L1

(50 it.)

Nir et al. [7]

AAE 32.43° 2.85° 0.85°

Frames per second

[1] B.K. Horn and B.G. Schunck. Determinig Optical Flow. Artificial Intelligence, 1981[7] T. Nir and A.M. Bruckstein and R. Kimmel, Over-Parameterized Variational Optical Flow, IJCV 2007



ICG Conclusion & Future work

• We have developed a duality based algorithm for TV-L1 optical flow computation

• We have implemented this algorithm on state-of-the-art graphics hardware

• In summary, we obtained an optical flow algorithm having a realtime performance of ~45 fps for 512x512 images

• Implementation in CUDA should give an additional speedup

• More sophisticated data terms for illumination changes• Multigrid techniques for the dual formulation



ICG

Demo



ICG Solution of the ROF model

• Compute the minimizer of ROF model– Solution of a huge sytem of non-linear equations– Leads to iterative algorithms

• Primal formulation:– Fixed-point scheme of Vogel and Oman [6]

• Dual formulation– Fixed-point scheme of Chambolle [5]

,...3,2,1 ,01 1

1

nvuu

u nn

n

,...3,2,1 ,01 nvpvpp nnn

[5] A. Chambolle, An Algorithm for Total Variation Minimization and Applications, 2004[6] C. R. Vogel and M. E. Oman. Iterative Methods For Total Variation Denoising. 1996

professor horst cerjak, 19.12.2005 1 thomas pock a duality based approach for realtime tv-l 1...

Documents

optical flow icg tv

realtime tv

determinig optical flow

thomas pock

normal flow

professor horst cerjak

optical flow christopher

constraint tv