dagm 2011 tutorial on convex optimization for …dagm 2011 tutorial on convex optimization for...
TRANSCRIPT
![Page 1: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/1.jpg)
DAGM 2011 Tutorial onConvex Optimization for Computer Vision
Part 3: Convex Solutions for Stereo and Optical Flow
Daniel CremersComputer Vision Group
Technical University of Munich
Graz University of Technology
Thomas PockInstitute for Computer Graphics and Vision
Graz University of Technology
Frankfurt, August 30, 2011
![Page 2: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/2.jpg)
tugrazGraz University of Technology
Overview
1 Motion estimation
2 Stereo estimation
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
2 / 21
![Page 3: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/3.jpg)
tugrazGraz University of Technology
Motion estimation
Motion estimation (optical flow) is a central topic in computer vision,
Computes a 2D vector field, describing the motion of pixel intensities
Applications:
Tracking
Video compression, video interpolation
3D reconstruction
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
3 / 21
![Page 4: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/4.jpg)
tugrazGraz University of Technology
Motion estimation
Motion estimation (optical flow) is a central topic in computer vision,
Computes a 2D vector field, describing the motion of pixel intensities
Applications:
Tracking
Video compression, video interpolation
3D reconstruction
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
3 / 21
![Page 5: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/5.jpg)
tugrazGraz University of Technology
Motion estimation
Motion estimation (optical flow) is a central topic in computer vision,
Computes a 2D vector field, describing the motion of pixel intensities
Applications:
Tracking
Video compression, video interpolation
3D reconstruction
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
3 / 21
![Page 6: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/6.jpg)
tugrazGraz University of Technology
Motion estimation
Motion estimation (optical flow) is a central topic in computer vision,
Computes a 2D vector field, describing the motion of pixel intensities
Applications:
Tracking
Video compression, video interpolation
3D reconstruction
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
3 / 21
![Page 7: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/7.jpg)
tugrazGraz University of Technology
Challenges
Motion estimation is still a very difficult problem
Aperture problem
No information in untextured areas
Illumination changes, shadows, ...
Large motion of small objects, occlusions, ...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 21
![Page 8: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/8.jpg)
tugrazGraz University of Technology
Challenges
Motion estimation is still a very difficult problem
Aperture problem
No information in untextured areas
Illumination changes, shadows, ...
Large motion of small objects, occlusions, ...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 21
![Page 9: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/9.jpg)
tugrazGraz University of Technology
Challenges
Motion estimation is still a very difficult problem
Aperture problem
No information in untextured areas
Illumination changes, shadows, ...
Large motion of small objects, occlusions, ...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 21
![Page 10: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/10.jpg)
tugrazGraz University of Technology
Challenges
Motion estimation is still a very difficult problem
Aperture problem
No information in untextured areas
Illumination changes, shadows, ...
Large motion of small objects, occlusions, ...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 21
![Page 11: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/11.jpg)
tugrazGraz University of Technology
Challenges
Motion estimation is still a very difficult problem
Aperture problem
No information in untextured areas
Illumination changes, shadows, ...
Large motion of small objects, occlusions, ...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 21
![Page 12: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/12.jpg)
tugrazGraz University of Technology
Challenges
Motion estimation is still a very difficult problem
Aperture problem
No information in untextured areas
Illumination changes, shadows, ...
Large motion of small objects, occlusions, ...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 21
![Page 13: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/13.jpg)
tugrazGraz University of Technology
Challenges
Motion estimation is still a very difficult problem
Aperture problem
No information in untextured areas
Illumination changes, shadows, ...
Large motion of small objects, occlusions, ...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 21
![Page 14: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/14.jpg)
tugrazGraz University of Technology
Challenges
Motion estimation is still a very difficult problem
Aperture problem
No information in untextured areas
Illumination changes, shadows, ...
Large motion of small objects, occlusions, ...
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
4 / 21
![Page 15: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/15.jpg)
tugrazGraz University of Technology
The correspondence problem
Find corresponding points in successive frames
Brightness (color) constancy assumption
I1(x)− I2(x + u(x)) ≈ 0
u(x) = (u1(x), u2(x)) is the displacement vector
Ambiguity: Many points with similar brightness (color)!
Generalization: Constancy of image features (gradients, NCC, ...)
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
5 / 21
![Page 16: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/16.jpg)
tugrazGraz University of Technology
Variational motion estimation
Generic variational model for motion estimation
minu
R(u)︸ ︷︷ ︸Regularization term
+
∫Ω|I1(x)− I2(x + u(x))|p dx︸ ︷︷ ︸
Data term
Regularization term:Should favor physically meaningful flow fieldsPopular convex regularizers: Quadratic, total variation, ...
Data term:Highly non-convex → hard to minimizeDifferent strategies to deal with the non-convexity of the data term
Vast literature on motion estimation:Window based optical flow: [Lucas, Kanade, 1981]Variational optical flow: [Horn, Schunck, 1981]Discontinuity preserving optical flow: [Shulman, Herve ’89]Robust optical flow: [Black, Anadan, ’93]Highly accurate optical flow: [Brox, Bruhn, Papenberg, Weickert ’04]Real-time optical flow: [A. Bruhn, J. Weickert, T. Kohlberger, C. Schnorr ’05]Primal-dual optimization on the GPU: [Zach, Pock, Bischof ’07]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
6 / 21
![Page 17: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/17.jpg)
tugrazGraz University of Technology
Variational motion estimation
Generic variational model for motion estimation
minu
R(u)︸ ︷︷ ︸Regularization term
+
∫Ω|I1(x)− I2(x + u(x))|p dx︸ ︷︷ ︸
Data term
Regularization term:Should favor physically meaningful flow fieldsPopular convex regularizers: Quadratic, total variation, ...
Data term:Highly non-convex → hard to minimizeDifferent strategies to deal with the non-convexity of the data term
Vast literature on motion estimation:Window based optical flow: [Lucas, Kanade, 1981]Variational optical flow: [Horn, Schunck, 1981]Discontinuity preserving optical flow: [Shulman, Herve ’89]Robust optical flow: [Black, Anadan, ’93]Highly accurate optical flow: [Brox, Bruhn, Papenberg, Weickert ’04]Real-time optical flow: [A. Bruhn, J. Weickert, T. Kohlberger, C. Schnorr ’05]Primal-dual optimization on the GPU: [Zach, Pock, Bischof ’07]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
6 / 21
![Page 18: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/18.jpg)
tugrazGraz University of Technology
Variational motion estimation
Generic variational model for motion estimation
minu
R(u)︸ ︷︷ ︸Regularization term
+
∫Ω|I1(x)− I2(x + u(x))|p dx︸ ︷︷ ︸
Data term
Regularization term:Should favor physically meaningful flow fieldsPopular convex regularizers: Quadratic, total variation, ...
Data term:Highly non-convex → hard to minimizeDifferent strategies to deal with the non-convexity of the data term
Vast literature on motion estimation:Window based optical flow: [Lucas, Kanade, 1981]Variational optical flow: [Horn, Schunck, 1981]Discontinuity preserving optical flow: [Shulman, Herve ’89]Robust optical flow: [Black, Anadan, ’93]Highly accurate optical flow: [Brox, Bruhn, Papenberg, Weickert ’04]Real-time optical flow: [A. Bruhn, J. Weickert, T. Kohlberger, C. Schnorr ’05]Primal-dual optimization on the GPU: [Zach, Pock, Bischof ’07]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
6 / 21
![Page 19: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/19.jpg)
tugrazGraz University of Technology
Variational motion estimation
Generic variational model for motion estimation
minu
R(u)︸ ︷︷ ︸Regularization term
+
∫Ω|I1(x)− I2(x + u(x))|p dx︸ ︷︷ ︸
Data term
Regularization term:Should favor physically meaningful flow fieldsPopular convex regularizers: Quadratic, total variation, ...
Data term:Highly non-convex → hard to minimizeDifferent strategies to deal with the non-convexity of the data term
Vast literature on motion estimation:Window based optical flow: [Lucas, Kanade, 1981]Variational optical flow: [Horn, Schunck, 1981]Discontinuity preserving optical flow: [Shulman, Herve ’89]Robust optical flow: [Black, Anadan, ’93]Highly accurate optical flow: [Brox, Bruhn, Papenberg, Weickert ’04]Real-time optical flow: [A. Bruhn, J. Weickert, T. Kohlberger, C. Schnorr ’05]Primal-dual optimization on the GPU: [Zach, Pock, Bischof ’07]
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
6 / 21
![Page 20: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/20.jpg)
tugrazGraz University of Technology
Linearization of the image
Perform a first order Taylor expansion of the function I2(x + u(x)) at x + u0(x)[Horn, Schunck, 1981], [Lucas, Kanade, 1981]
I2(x + u(x)) ≈ I2(x + u0(x)) + 〈∇I2(x + u0(x)), u(x)− u0(x)〉Only valid close to u0, i.e. ‖u(x)− u0(x)‖ ≤ ε
Leads to the classical optical flow constraint:
ρ(u) = I1(x)− I2(x + u0(x))− 〈∇I2(x + u0(x)), u(x)− u0(x)〉 ≈ 0
Note: ρ(u) is linear in u and hence |ρ(u)| is convex!
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
7 / 21
![Page 21: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/21.jpg)
tugrazGraz University of Technology
Linearization of the image
Perform a first order Taylor expansion of the function I2(x + u(x)) at x + u0(x)[Horn, Schunck, 1981], [Lucas, Kanade, 1981]
I2(x + u(x)) ≈ I2(x + u0(x)) + 〈∇I2(x + u0(x)), u(x)− u0(x)〉Only valid close to u0, i.e. ‖u(x)− u0(x)‖ ≤ ε
Leads to the classical optical flow constraint:
ρ(u) = I1(x)− I2(x + u0(x))− 〈∇I2(x + u0(x)), u(x)− u0(x)〉 ≈ 0
Note: ρ(u) is linear in u and hence |ρ(u)| is convex!
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
7 / 21
![Page 22: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/22.jpg)
tugrazGraz University of Technology
Linearization of the image
Perform a first order Taylor expansion of the function I2(x + u(x)) at x + u0(x)[Horn, Schunck, 1981], [Lucas, Kanade, 1981]
I2(x + u(x)) ≈ I2(x + u0(x)) + 〈∇I2(x + u0(x)), u(x)− u0(x)〉Only valid close to u0, i.e. ‖u(x)− u0(x)‖ ≤ ε
Leads to the classical optical flow constraint:
ρ(u) = I1(x)− I2(x + u0(x))− 〈∇I2(x + u0(x)), u(x)− u0(x)〉 ≈ 0
Note: ρ(u) is linear in u and hence |ρ(u)| is convex!
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
7 / 21
![Page 23: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/23.jpg)
tugrazGraz University of Technology
TV-L1 motion estimation
It turns out that total variation regularization in combination with a L1 data termperforms well
Total variation allows for motion discontinuities
L1 data term allows for outliers in the data term (occlusions, noise, ...)
min‖u−u0‖≤ε
α
∫Ω|Du|+ ‖ρ(u)‖1
Non-differentiable and hence difficult to solve
Smoothing and fixed-point iteration: [Brox, Bruhn, Papenberg, Weickert ’04]
Primal-dual optimization: [Chambolle, Pock, ’10]
min‖u−u0‖≤ε
max‖p‖∞≤α
−∫
Ωu div p dx + ‖ρ(u)‖1
Allows to compute the exact solution
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
8 / 21
![Page 24: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/24.jpg)
tugrazGraz University of Technology
TV-L1 motion estimation
It turns out that total variation regularization in combination with a L1 data termperforms well
Total variation allows for motion discontinuities
L1 data term allows for outliers in the data term (occlusions, noise, ...)
min‖u−u0‖≤ε
α
∫Ω|Du|+ ‖ρ(u)‖1
Non-differentiable and hence difficult to solve
Smoothing and fixed-point iteration: [Brox, Bruhn, Papenberg, Weickert ’04]
Primal-dual optimization: [Chambolle, Pock, ’10]
min‖u−u0‖≤ε
max‖p‖∞≤α
−∫
Ωu div p dx + ‖ρ(u)‖1
Allows to compute the exact solution
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
8 / 21
![Page 25: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/25.jpg)
tugrazGraz University of Technology
Second-order approximation of the data term
Consider a more general non-convex data term of the form∫Ωφ(x , u(x)) dx
Perform a second order Taylor expansion of the data term φ(x , u(x)) aroundu0(x) [Werlberger, Pock, Bischof ’10]
φ(x , u(x)) ≈ φ(x , u0(x)) + (∇φ(x , u0(x)))T (u(x)− u0(x)) +
(u(x)− u0(x))T(∇2φ(x , u0(x))
)(u(x)− u0(x)) ,
To ensure convexity the Hessian ∇2φ(x , u0(x)) has to be positive semidefinite
We use the following diagonal approximation of the Hessian
∇2φ =
[(φxx (x , u0(x)))+ 0
0 (φyy (x , u0(x)))+
]
Can be used with arbitrary data terms: SAD, NCC, ...
Still only valid in a small neighborhood around u0
Minimization using primal-dual schemes
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
9 / 21
![Page 26: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/26.jpg)
tugrazGraz University of Technology
Large displacements
How can we compute large displacements?
Integrate the algorithm in a coarse-to fine / warping framework
Similar to multigrid schemes, speeds up the minimization process
Does not give any guarantees!
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
10 / 21
![Page 27: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/27.jpg)
tugrazGraz University of Technology
Large displacement optical flow without warping
Consider the following equivalent generic formulation [Steinbrucker, Pock,Cremers, ’09]
minuR(u) +
∫Ωφ(u) dx ⇐⇒ min
u,vR(u) +
∫Ωφ(v) dx s.t. u = v
Quadratic penality approach to obtain a unconstrained formulation
minu,vR(u) +
1
2θ‖u− v‖2
2 +
∫Ωφ(v) dx
Becomes equivalent to the constrained formulation for θ → 0+
Observations:Solution with respect to u reduces to an image denoising problemSolution with respect to v reduces to pointwise non-convex problems
Annealing-type scheme: Alternating minimization for a sequence of decreasingparameters θi
Advantages: No coarse-to-fine, no warping, arbitrary data terms
Disadvantage: Results strongly depend on the sequence θi
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
11 / 21
![Page 28: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/28.jpg)
tugrazGraz University of Technology
Large displacement optical flow without warping
Consider the following equivalent generic formulation [Steinbrucker, Pock,Cremers, ’09]
minuR(u) +
∫Ωφ(u) dx ⇐⇒ min
u,vR(u) +
∫Ωφ(v) dx s.t. u = v
Quadratic penality approach to obtain a unconstrained formulation
minu,vR(u) +
1
2θ‖u− v‖2
2 +
∫Ωφ(v) dx
Becomes equivalent to the constrained formulation for θ → 0+
Observations:Solution with respect to u reduces to an image denoising problemSolution with respect to v reduces to pointwise non-convex problems
Annealing-type scheme: Alternating minimization for a sequence of decreasingparameters θi
Advantages: No coarse-to-fine, no warping, arbitrary data terms
Disadvantage: Results strongly depend on the sequence θi
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
11 / 21
![Page 29: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/29.jpg)
tugrazGraz University of Technology
Large displacement optical flow without warping
Consider the following equivalent generic formulation [Steinbrucker, Pock,Cremers, ’09]
minuR(u) +
∫Ωφ(u) dx ⇐⇒ min
u,vR(u) +
∫Ωφ(v) dx s.t. u = v
Quadratic penality approach to obtain a unconstrained formulation
minu,vR(u) +
1
2θ‖u− v‖2
2 +
∫Ωφ(v) dx
Becomes equivalent to the constrained formulation for θ → 0+
Observations:Solution with respect to u reduces to an image denoising problemSolution with respect to v reduces to pointwise non-convex problems
Annealing-type scheme: Alternating minimization for a sequence of decreasingparameters θi
Advantages: No coarse-to-fine, no warping, arbitrary data terms
Disadvantage: Results strongly depend on the sequence θi
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
11 / 21
![Page 30: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/30.jpg)
tugrazGraz University of Technology
Large displacement optical flow without warping
Consider the following equivalent generic formulation [Steinbrucker, Pock,Cremers, ’09]
minuR(u) +
∫Ωφ(u) dx ⇐⇒ min
u,vR(u) +
∫Ωφ(v) dx s.t. u = v
Quadratic penality approach to obtain a unconstrained formulation
minu,vR(u) +
1
2θ‖u− v‖2
2 +
∫Ωφ(v) dx
Becomes equivalent to the constrained formulation for θ → 0+
Observations:Solution with respect to u reduces to an image denoising problemSolution with respect to v reduces to pointwise non-convex problems
Annealing-type scheme: Alternating minimization for a sequence of decreasingparameters θi
Advantages: No coarse-to-fine, no warping, arbitrary data terms
Disadvantage: Results strongly depend on the sequence θi
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
11 / 21
![Page 31: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/31.jpg)
tugrazGraz University of Technology
Varying illumination
Note: optical flow 6= motion estimation!
Instead of (or in combination with) the original images I1,2 use gradient images∇I1,2 [Brox, Bruhn, Papenberg, Weickert ’04]
Photometric invariants: [Mileva, Bruhn, Weickert ’07]
Structure-texture decomposition: Illumination changes and shadows correspondto large image features [Wedel, Pock, Zach, Bischof, Cremers ’08]
Addititve decomposition using the ROF model:
I = S + T , S := arg minu
TV(u) +λ
2‖u − I‖2
2
(a) I (b) S (c) T
Use texture component T to compute the optical flow
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
12 / 21
![Page 32: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/32.jpg)
tugrazGraz University of Technology
Varying illumination
Note: optical flow 6= motion estimation!
Instead of (or in combination with) the original images I1,2 use gradient images∇I1,2 [Brox, Bruhn, Papenberg, Weickert ’04]
Photometric invariants: [Mileva, Bruhn, Weickert ’07]
Structure-texture decomposition: Illumination changes and shadows correspondto large image features [Wedel, Pock, Zach, Bischof, Cremers ’08]
Addititve decomposition using the ROF model:
I = S + T , S := arg minu
TV(u) +λ
2‖u − I‖2
2
(a) I (b) S (c) T
Use texture component T to compute the optical flow
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
12 / 21
![Page 33: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/33.jpg)
tugrazGraz University of Technology
Varying illumination
Note: optical flow 6= motion estimation!
Instead of (or in combination with) the original images I1,2 use gradient images∇I1,2 [Brox, Bruhn, Papenberg, Weickert ’04]
Photometric invariants: [Mileva, Bruhn, Weickert ’07]
Structure-texture decomposition: Illumination changes and shadows correspondto large image features [Wedel, Pock, Zach, Bischof, Cremers ’08]
Addititve decomposition using the ROF model:
I = S + T , S := arg minu
TV(u) +λ
2‖u − I‖2
2
(a) I (b) S (c) T
Use texture component T to compute the optical flow
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
12 / 21
![Page 34: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/34.jpg)
tugrazGraz University of Technology
Varying illumination
Note: optical flow 6= motion estimation!
Instead of (or in combination with) the original images I1,2 use gradient images∇I1,2 [Brox, Bruhn, Papenberg, Weickert ’04]
Photometric invariants: [Mileva, Bruhn, Weickert ’07]
Structure-texture decomposition: Illumination changes and shadows correspondto large image features [Wedel, Pock, Zach, Bischof, Cremers ’08]
Addititve decomposition using the ROF model:
I = S + T , S := arg minu
TV(u) +λ
2‖u − I‖2
2
(a) I (b) S (c) T
Use texture component T to compute the optical flow
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
12 / 21
![Page 35: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/35.jpg)
tugrazGraz University of Technology
Varying illumination
Note: optical flow 6= motion estimation!
Instead of (or in combination with) the original images I1,2 use gradient images∇I1,2 [Brox, Bruhn, Papenberg, Weickert ’04]
Photometric invariants: [Mileva, Bruhn, Weickert ’07]
Structure-texture decomposition: Illumination changes and shadows correspondto large image features [Wedel, Pock, Zach, Bischof, Cremers ’08]
Addititve decomposition using the ROF model:
I = S + T , S := arg minu
TV(u) +λ
2‖u − I‖2
2
(a) I (b) S (c) T
Use texture component T to compute the optical flow
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
12 / 21
![Page 36: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/36.jpg)
tugrazGraz University of Technology
Modified optical flow constraint
Recall the optical flow constraint
ρ(u) = I1(x)− I2(x + u0(x))− 〈∇I2(x + u0(x)), u(x)− u0(x)〉 ≈ 0
We can modify the constraint [Shulman, Herve ’89]
δ(u, v) = I1(x)− I2(x + u0(x))− 〈∇I2(x + u0(x)), u(x)− u0(x)〉 − v(x) ≈ 0
v(x) is a smooth function modeling illumination changesNote that δ(u, v) is still linear in u and v !
Additional regularization needed for v(x)
min‖u−u0‖≤ε,v
α
∫Ω|Du|+ β
∫Ω|Dv |+ ‖δ(u(x), v(x))‖1
(a) Input (b) Ground truth (c) Estimated motion (d) Illumination
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
13 / 21
![Page 37: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/37.jpg)
tugrazGraz University of Technology
Modified optical flow constraint
Recall the optical flow constraint
ρ(u) = I1(x)− I2(x + u0(x))− 〈∇I2(x + u0(x)), u(x)− u0(x)〉 ≈ 0
We can modify the constraint [Shulman, Herve ’89]
δ(u, v) = I1(x)− I2(x + u0(x))− 〈∇I2(x + u0(x)), u(x)− u0(x)〉 − v(x) ≈ 0
v(x) is a smooth function modeling illumination changesNote that δ(u, v) is still linear in u and v !
Additional regularization needed for v(x)
min‖u−u0‖≤ε,v
α
∫Ω|Du|+ β
∫Ω|Dv |+ ‖δ(u(x), v(x))‖1
(a) Input (b) Ground truth (c) Estimated motion (d) Illumination
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
13 / 21
![Page 38: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/38.jpg)
tugrazGraz University of Technology
Modified optical flow constraint
Recall the optical flow constraint
ρ(u) = I1(x)− I2(x + u0(x))− 〈∇I2(x + u0(x)), u(x)− u0(x)〉 ≈ 0
We can modify the constraint [Shulman, Herve ’89]
δ(u, v) = I1(x)− I2(x + u0(x))− 〈∇I2(x + u0(x)), u(x)− u0(x)〉 − v(x) ≈ 0
v(x) is a smooth function modeling illumination changesNote that δ(u, v) is still linear in u and v !
Additional regularization needed for v(x)
min‖u−u0‖≤ε,v
α
∫Ω|Du|+ β
∫Ω|Dv |+ ‖δ(u(x), v(x))‖1
(a) Input (b) Ground truth (c) Estimated motion (d) Illumination
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
13 / 21
![Page 39: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/39.jpg)
tugrazGraz University of Technology
Overview
1 Motion estimation
2 Stereo estimation
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
14 / 21
![Page 40: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/40.jpg)
tugrazGraz University of Technology
StereoIf I1 and I2 come from a stereo camera or a moving camera that browses a staticscene, the displacement can be restricted to 1D problems on the epipolar lines,[Slesareva, Bruhn, Weickert ’05]Each stereo pair can be normalized such that the displacement is only horizontallyThe depth z can be computed from the displacement u via
z(x , y) =bf
u(x , y)
where b is the baseline and f is the focal length of the camera
Optical flow constraint for stereo
ρ(u) = I1 − I2(x + u0(x , y), y)− ∂x I2(x + u0(x , y), y)(u(x , y)− u0(x , y)) ≈ 0
TV-L1 based stereo
min‖u−u0‖≤ε
α
∫Ω|Du|+ ‖ρ(u(x , y))‖1
AdvantagesHighly accurate due to sub-pixel accuracyfast to compute (real-time)
DisadvantagesDoes not compute the globally optimal solution (coarse-to-fine)Large baseline leads to more accurate results but causes large displacements
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
15 / 21
![Page 41: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/41.jpg)
tugrazGraz University of Technology
StereoIf I1 and I2 come from a stereo camera or a moving camera that browses a staticscene, the displacement can be restricted to 1D problems on the epipolar lines,[Slesareva, Bruhn, Weickert ’05]Each stereo pair can be normalized such that the displacement is only horizontallyThe depth z can be computed from the displacement u via
z(x , y) =bf
u(x , y)
where b is the baseline and f is the focal length of the camera
Optical flow constraint for stereo
ρ(u) = I1 − I2(x + u0(x , y), y)− ∂x I2(x + u0(x , y), y)(u(x , y)− u0(x , y)) ≈ 0
TV-L1 based stereo
min‖u−u0‖≤ε
α
∫Ω|Du|+ ‖ρ(u(x , y))‖1
AdvantagesHighly accurate due to sub-pixel accuracyfast to compute (real-time)
DisadvantagesDoes not compute the globally optimal solution (coarse-to-fine)Large baseline leads to more accurate results but causes large displacements
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
15 / 21
![Page 42: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/42.jpg)
tugrazGraz University of Technology
StereoIf I1 and I2 come from a stereo camera or a moving camera that browses a staticscene, the displacement can be restricted to 1D problems on the epipolar lines,[Slesareva, Bruhn, Weickert ’05]Each stereo pair can be normalized such that the displacement is only horizontallyThe depth z can be computed from the displacement u via
z(x , y) =bf
u(x , y)
where b is the baseline and f is the focal length of the camera
Optical flow constraint for stereo
ρ(u) = I1 − I2(x + u0(x , y), y)− ∂x I2(x + u0(x , y), y)(u(x , y)− u0(x , y)) ≈ 0
TV-L1 based stereo
min‖u−u0‖≤ε
α
∫Ω|Du|+ ‖ρ(u(x , y))‖1
AdvantagesHighly accurate due to sub-pixel accuracyfast to compute (real-time)
DisadvantagesDoes not compute the globally optimal solution (coarse-to-fine)Large baseline leads to more accurate results but causes large displacementsDaniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
15 / 21
![Page 43: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/43.jpg)
tugrazGraz University of Technology
Application: range estimation in a driving car (with Daimler AG)
Input images provided by a calibrated stereo rig
(a) Left image (b) Right image
Range image computed by the TV-L1 based stereo algorithm
(a) Range image
0 20 40 60 80 100 120 14060
80
100
120
140
160
180
200
220
240
Distance along profile
(b) Profile of street
Total variation regularization leads to the staircasing effect!
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
16 / 21
![Page 44: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/44.jpg)
tugrazGraz University of Technology
Total generalized variation
The total variation can be written (via the convex conjugate) as
TVα(u) = α
∫Ω|Du| = sup
∫Ωu div v dx
∣∣∣ v ∈ C1c(Ω,Rd ), ‖v‖∞ ≤ α
,
In [Bredies, Kunisch, Pock, SIIMS’10], we proposed a generalization of the totalvariation to higher order smoothness.
TGVkα(u) = sup
∫Ωu divk v dx
∣∣∣ v ∈ Ckc (Ω,Symk (Rd )),
‖divl v‖∞ ≤ αl , l = 0, . . . , k − 1,
For k = 2 it can be written as
TGV2α(u) = inf
wα1
∫Ω|Du − w|+ α0
∫Ω|Dw|
TGV2 can be used to reconstruct piecewise affine functions
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
17 / 21
![Page 45: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/45.jpg)
tugrazGraz University of Technology
Total generalized variation
The total variation can be written (via the convex conjugate) as
TVα(u) = α
∫Ω|Du| = sup
∫Ωu div v dx
∣∣∣ v ∈ C1c(Ω,Rd ), ‖v‖∞ ≤ α
,
In [Bredies, Kunisch, Pock, SIIMS’10], we proposed a generalization of the totalvariation to higher order smoothness.
TGVkα(u) = sup
∫Ωu divk v dx
∣∣∣ v ∈ Ckc (Ω,Symk (Rd )),
‖divl v‖∞ ≤ αl , l = 0, . . . , k − 1,
For k = 2 it can be written as
TGV2α(u) = inf
wα1
∫Ω|Du − w|+ α0
∫Ω|Dw|
TGV2 can be used to reconstruct piecewise affine functions
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
17 / 21
![Page 46: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/46.jpg)
tugrazGraz University of Technology
Total generalized variation
The total variation can be written (via the convex conjugate) as
TVα(u) = α
∫Ω|Du| = sup
∫Ωu div v dx
∣∣∣ v ∈ C1c(Ω,Rd ), ‖v‖∞ ≤ α
,
In [Bredies, Kunisch, Pock, SIIMS’10], we proposed a generalization of the totalvariation to higher order smoothness.
TGVkα(u) = sup
∫Ωu divk v dx
∣∣∣ v ∈ Ckc (Ω,Symk (Rd )),
‖divl v‖∞ ≤ αl , l = 0, . . . , k − 1,
For k = 2 it can be written as
TGV2α(u) = inf
wα1
∫Ω|Du − w|+ α0
∫Ω|Dw|
TGV2 can be used to reconstruct piecewise affine functions
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
17 / 21
![Page 47: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/47.jpg)
tugrazGraz University of Technology
Image restoration examples
(a) Clean image (b) Noisy image
(c) TV (d) TGV2
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
18 / 21
![Page 48: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/48.jpg)
tugrazGraz University of Technology
Image restoration examples
0
10
20
30
40
50
60
010
2030
4050
60
0
0.2
0.4
0.6
0.8
1
(a) TV
0
10
20
30
40
50
60
010
2030
4050
60
0
0.2
0.4
0.6
0.8
1
(b) TGV2
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
18 / 21
![Page 49: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/49.jpg)
tugrazGraz University of Technology
TGV based stereo
Simply replace TV regularization by TGV regularization in the stereo model[Ranftl, Pock, Gehrig, Franke ’11]
min‖u−u0‖≤ε,w
α1
∫Ω|Du − w|+ α0
∫Ω|Dw|+ ‖ρ(u(x , y))‖1
Comparison on the stereo problem
0 20 40 60 80 100 120 14060
80
100
120
140
160
180
200
220
240
Distance along profile
(a) TV
0 20 40 60 80 100 120 140 1600
50
100
150
200
250
Distance along profile
(b) TGV2
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
19 / 21
![Page 50: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/50.jpg)
tugrazGraz University of Technology
Range estimation from a driving car
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
20 / 21
![Page 51: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/51.jpg)
tugrazGraz University of Technology
Summary and open questions
Introduced the problem of motion estimation in computer vision
Motion estimation is still a challenging problem, not near to be solved
Highly non-convex data term leads to numerical difficulties
A simple linearization approach works well in practice
Can be used for stereo estimation
TGV regularization avoids staircasing-artifacts
Global Solutions for Motion and Stereo?
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
21 / 21
![Page 52: DAGM 2011 Tutorial on Convex Optimization for …DAGM 2011 Tutorial on Convex Optimization for Computer Vision Part 3: Convex Solutions for Stereo and Optical Flow Daniel Cremers Computer](https://reader035.vdocument.in/reader035/viewer/2022070807/5f0611457e708231d416208a/html5/thumbnails/52.jpg)
tugrazGraz University of Technology
Summary and open questions
Introduced the problem of motion estimation in computer vision
Motion estimation is still a challenging problem, not near to be solved
Highly non-convex data term leads to numerical difficulties
A simple linearization approach works well in practice
Can be used for stereo estimation
TGV regularization avoids staircasing-artifacts
Global Solutions for Motion and Stereo?
Daniel Cremers and Thomas Pock Frankfurt, August 30, 2011 Convex Optimization for Computer Vision
21 / 21