![Page 1: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/1.jpg)
Fast and Accurate Optical Flow Estimation
Primal-Dual Schemes andSecond Order Priors
Thomas Pock and Daniel CremersCVPR Group, University of Bonn
Collaborators: Christopher Zach, Markus Unger, Werner Trobin, and Horst Bischof
![Page 2: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/2.jpg)
Variational Optical Flow – Short History
Horn and Schunck
1981
Black and Anadan, Cohen
1993
Aubert
2000 2004
Brox et al.
2006
Bruhn et al.
![Page 3: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/3.jpg)
![Page 4: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/4.jpg)
Outline• Model of Horn and Schunck• TV-L1 Model• Fast Numerical Scheme• Parallel Implementation• 2nd order Prior
![Page 5: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/5.jpg)
The Model of Horn and Schunck [1]
Regularization Term Data Term (OFC)
+ Convex+ Easy to solve- Does not allow for sharp edges in the solution- Sensitive to outliers violating the OFC
[1] Horn and Schunck. Determinig Optical Flow. Artificial Intelligence, 1981
![Page 6: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/6.jpg)
Can we do better?
• Replace quadratic functions by L1 – norms• Done by Cohen, Aubert, Brox, Bruhn, ...
+Allows for discontinuities in the flow field+Robust to some extent to outliers in the OFC+Still convex- Much harder to solve
![Page 7: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/7.jpg)
How can we minimize this functional ?
• Compute Euler-Lagrange Equations
• Non-linear, non-smooth, ...
![Page 8: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/8.jpg)
Standard Approach
• Replace L1 – norm by regularized variants (Charbonnier function)
• Example:• Small epsilon: Nearly degenerated• Large espilon: Smears edges
![Page 9: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/9.jpg)
Our Approach(1)
• Introduce auxiliary variables and constraints
• Quadratic penalty
![Page 10: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/10.jpg)
Our Approach(2)• What do we gain?• We solve a sequence of simpler problems
Algorithm[3]:1. For fixed (u´,v´), solve for(u,v) using Chambolle‘s algorithm[4]2. For fixed (u,v), solve for (u´,v´) using a 1D shrinkage formula3. Goto 1. until convergence
[2] Rudin, Osher and Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, 1992[3] Zach, Pock and Bischof. A Duality Based Algorithm for Realtime TV-L1 Optical Flow, DAGM 2007 [4] Chambolle. An Algorithm for Total Variation Minimization, 2004.
1D Problem
ROF Model [2]
![Page 11: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/11.jpg)
Implementation
• Numerical scheme can be easily parallelized• We use state-of-the-art GPUs
![Page 12: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/12.jpg)
Performance Evaluation
Image Size Frames per Second128x128 192256x256 108512x512 36
• TV-L1 Optical Flow Implemented in CUDA 2.0• Computed on Nvidia GeForce GTX 280• 25 Overall Iterations (5 Chambolle Iterations)
![Page 13: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/13.jpg)
Results for TV-L1
Ground Truth:
Our Results:
Input Image:
![Page 14: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/14.jpg)
2nd order Prior• TV regularization favors piecewise constant flow
fields (frontoparallel motion)• Extension to piecewise affine flow fields?• Approach of Cremers et al. [5]– Fixed number of regions
• Approach of Nir et al. [6]– Over-parametrized optical flow
• Our approach [7]– 2nd order derivatives to regularize flow field[5] Cremers and Soatto, Motion Competition: A Variational Framework for Piecwise Parametric Motion
Segmentation.[6] Nir, Bruckstein and Kimmel, Over-Parameterized Variational Optical Flow, IJCV 2007[7] Trobin, Pock, Cremers and Bischof, An Unbiased Second-Order prPior for High-AccuracyMotion Estimation, DAGM 2008
![Page 15: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/15.jpg)
2nd-L1 Optical Flow
• 2nd order derivatives are not orthogonal• We use a transformation due to Danielsson [8]
• Optimization– Similar strategy to TV-L1
– 4th order PDE
[8] Danielsson and Lin, Efficient Detection of Second-Degree Variations in 2D and 3D Images, 2001.
![Page 16: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/16.jpg)
Comparison
Ground truth TV-L1 2nd -L1
![Page 17: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/17.jpg)
Results for 2nd-L1
Ground Truth:
Our Results:
![Page 18: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/18.jpg)
Conclusion
• TV-L1 Optical Flow– Fast Numerical Scheme
• Parallel Implementation– Realtime Performance
• 2nd order prior – Piecewise affine motion
![Page 19: Fast and Accurate Optical Flow Estimation Primal-Dual Schemes and Second Order Priors Thomas Pock and Daniel Cremers CVPR Group, University of Bonn Collaborators:](https://reader035.vdocument.in/reader035/viewer/2022062619/55199b7855034643068b49cc/html5/thumbnails/19.jpg)
Recent Application: Tracking