surface stereo with soft segmentation michael bleyer 1, carsten rother 2, pushmeet kohli 2 1 vienna...
TRANSCRIPT
![Page 1: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/1.jpg)
Surface Stereo with Soft Segmentation
Michael Bleyer1, Carsten Rother2, Pushmeet Kohli2
1Vienna University of Technology, Austria2Microsoft Research Cambridge, UK
ICVSS 2010
![Page 2: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/2.jpg)
Dense Stereo Matching
(Left Image) (Right Image)
![Page 3: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/3.jpg)
Dense Stereo Matching
(Left Image) (Right Image)
(Disparity Map)
![Page 4: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/4.jpg)
Common Approaches
Reference image
![Page 5: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/5.jpg)
Common Approaches
Reference image Disparity map
Assign pixels to disparity values
![Page 6: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/6.jpg)
Our Approach
Reference image
![Page 7: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/7.jpg)
Our Approach
Reference image Surface map
Assign pixels to 3D surfaces
![Page 8: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/8.jpg)
Our Approach
Reference image Surface map Disparity map
Assign pixels to 3D surfaces
Surfaces implicitly define disparities
![Page 9: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/9.jpg)
Our Approach
Reference image Surface map Disparity map
Assign pixels to 3D surfaces
Surfaces implicitly define disparities
• Our approach simultaneously infers:1. Which surfaces are present in the scene2. Which pixels belong to which surface
![Page 10: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/10.jpg)
Energy• Search an assignment of pixels to surfaces that
minimizes an energy:
• Surfaces: planes or B-splines.
Data Term:Computes pixel dissimilarities;
Penalty for occluded pixels
Smoothness Term:Penalty on spatially neighboring pixels
assigned to different surfaces
Soft Segmentation Term:
Penalty on inconsistencies with a
given color segmentation
MDL Term:Penalty on the number of
surfaces
Curvature Term:Penalty on disparity
curvature
![Page 11: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/11.jpg)
Energy• Search an assignment of pixels to surfaces that
minimizes an energy:
• Surfaces: planes or B-splines.
Data Term:Computes pixel dissimilarities;
Penalty for occluded pixels
Smoothness Term:Penalty on spatially neighboring pixels
assigned to different surfaces
Soft Segmentation Term:
Penalty on inconsistencies with a
given color segmentation
MDL Term:Penalty on the number of
surfaces
Curvature Term:Penalty on disparity
curvature
Contributions
![Page 12: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/12.jpg)
Soft Segmentation Term• Common segmentation-based methods:
• Color segmentation of reference image• Assign each segment to a single surface• Fail if segment overlaps a disparity discontinuity.
Map reference image Ground truth disparities Result of hard segmentation method
![Page 13: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/13.jpg)
Soft Segmentation Term• Our approach:
• Prefer solutions consistent with a segmentation (lower energy).
• Segmentation = soft constraint.
Result of hard segmentation method
Our result
![Page 14: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/14.jpg)
Soft Segmentation Term
Segment
![Page 15: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/15.jpg)
Soft Segmentation Term
Segment
![Page 16: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/16.jpg)
Soft Segmentation Term
Segment
Subsegment
![Page 17: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/17.jpg)
Soft Segmentation Term
Segment
Subsegment
• Our term:• 0 penalty if all pixels within subsegment assigned to the same surface
• Constant penalty, otherwise
![Page 18: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/18.jpg)
Soft Segmentation Term
![Page 19: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/19.jpg)
Soft Segmentation Term
![Page 20: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/20.jpg)
MDL Term• Simple scene explanation better than unnecessarily
complex one. • Penalty on the number of surfaces. • Solution containing 5 surfaces cheaper than one with
100 surfaces.
Crop of the Cones image
Solution without our MDL term. Our MDL term.
![Page 21: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/21.jpg)
Curvature Term• Second order priors:
• Difficult to optimize in disparity-based representation due to triple cliques [Woodford et al., CVPR08].
• Our approach:• Curvature analytically computed from surface model.• Easy to optimize in surface-based representation
(unary term).
Result without curvature term. Result with curvature term.
![Page 22: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/22.jpg)
Improved Asymmetric Occlusion Handling• Uniqueness assumption violated for slanted surfaces:
• Several pixels of the same surface correspond to a single pixel of the second view.
• Our approach:• Pixels must not occlude each other if they lie on same
surface.• Avoids wrongly detected occlusions at slanted surfaces.
Standard Occlusion Handling Ours
![Page 23: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/23.jpg)
Energy Optimization• Not easy – label set of infinite size!• Fusion move approach [Lempitsky et al., ICCV07]:
Current Solution
![Page 24: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/24.jpg)
Energy Optimization• Not easy – label set of infinite size!• Fusion move approach [Lempitsky et al., ICCV07]:
Current Solution Proposal
![Page 25: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/25.jpg)
Energy Optimization• Not easy – label set of infinite size!• Fusion move approach [Lempitsky et al., ICCV07]:
Current Solution Proposal
Fusion Result
![Page 26: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/26.jpg)
Energy Optimization• Not easy – label set of infinite size!• Fusion move approach [Lempitsky et al., ICCV07]:
Current Solution Proposal
Fusion Result
• Computing the “optimal” fusion move:• Recent work on sparse
higher-order cliques ([Kohli et al., CVPR07] and [Rother et al., CVPR09]) for implementing soft segmentation term.
• Non-submodular energy optimized via QPBOI.
![Page 27: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/27.jpg)
Computed disparity maps
Disparity errors > 1 pixel
Assignment of pixels to surfaces
Left images with contour lines overlaid
![Page 28: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/28.jpg)
Computed disparity maps
Disparity errors > 1 pixel
Assignment of pixels to surfaces
Left images with contour lines overlaid
• 6th rank out of ~90 submissions in the Middlebury online table.
• 1th rank for the complex Teddy set on all error measures.
![Page 29: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/29.jpg)
![Page 30: Surface Stereo with Soft Segmentation Michael Bleyer 1, Carsten Rother 2, Pushmeet Kohli 2 1 Vienna University of Technology, Austria 2 Microsoft Research](https://reader035.vdocument.in/reader035/viewer/2022062519/56649e5d5503460f94b56b52/html5/thumbnails/30.jpg)
Conclusions
• Surface-based representation is important.• Enables several important contributions:
• Soft segmentation• MDL prior