bilateral and trilateral adaptive support weights in...
TRANSCRIPT
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Bilateral and Trilateral Adaptive SupportWeights in Stereo Vision
Ryan Beethe
Student, Colorado School of Mines
April 7, 2016
1 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Overview
1 Stereo MatchingImage ExampleBlock MatchingMatching Cost
2 Bilateral FilteringBilateral ExplanationAdaptive Support Weight
3 Trilateral FilteringTrilateral ExplanationTrilateral-based ASW
4 ASW In PracticeGPU ASWASW AlternativesTiming Comparison
2 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Stereo Matching
3 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Stereo Image Example [3]
Reference Image(left)
Target Image(right)
Disparity Image(left)
4 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Stereo Image ExampleHow Disparity Image is Generated
5 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Pixel MatchingAmbiguity issues
6 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Block MatchingBlocks of Pixels Reduce Ambiguity
7 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Block Matching
Left Image Disparity Image
8 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Matching Cost, Cost Aggregation
SAD = 2
SAD = 32
SAD = 17
9 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
SAD Limitation
What is wrong with using SAD on this reference window?
10 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Bilateral Filtering
11 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Bilateral ExplanationEdge-Preserving Version on Gaussian
Gaussian kernel (regardless of pixel values)
Bilateral “kernel” at a noisy step [4]
12 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Bilateral Example
Original Image Bilaterally Smoothed Image
13 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Adaptive Support WeightIntroduction
Reference Window Target Window 1 Target Window 2
14 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Adaptive Support WeightIndividual Support Weights
Reference Window Target Window 1 Target Window 2
Individual Window Support Weights
15 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Adaptive Support WeightCombined Support Weights
Reference Window
Target Window 1
Reference WindowSupport Weight
Target Window 1Support Weight
Combined SupportWeight
16 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Adaptive Support WeightCombined Support Weights, cont.
Reference Window
Target Window 2
Reference WindowSupport Weight
Target Window 2Support Weight
Combined SupportWeight
17 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Adaptive Support WeightImproved Stereo Output
Raw BM Output ASW Output[5]
18 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Other ASW Methods
Many alternative methods for adaptive support weight havebeen proposed based on the original bilateral-based ASWimplementation. Modifications include:
• Removing spacial component of bilateral filter
• Replacing bilateral with guided filter
• Replacing bilateral filter with some other filter
• Approximating the bilateral filter
• Calculating support weight for only the reference imageFor more discussion, see [2].
19 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Bilateral ASW Limitations
Image from [5].
20 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Trilateral Filtering
21 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Trilateral ExplanationTrilateral filter, in terms of Bilateral
22 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Trilateral ExplanationTrilateral filter, in terms of Bilateral, cont.
23 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Trilateral ExplanationCut-off Term
24 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Trilateral-based ASW
OriginalWindow
Bilateralsupportweight
Additionalsupportweight
term
Combinedsupportweight
25 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Trilateral-based ASW
Notice the trilateral-based ASW (left)outpreforms bilateral-based ASW (right)
near edges:
Images adapted from [1]
26 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Trilateral Performance
Image Algorithm nonocc all disc
Semi-Global BM 3.26 3.96 12.8Tsukuba Trilateral ASW 1.65 1.96 5.90
Bilateral ASW 1.38 1.85 6.90
Semi-Global BM 1.00 1.57 11.3Venus Trilateral ASW 0.14 0.31 1.51
Bilateral ASW 0.71 1.19 6.13
Semi-Global BM 6.02 12.2 16.3Teddy Trilateral ASW 6.25 11.8 15.1
Bilateral ASW 7.88 13.3 18.6
Semi-Global BM 3.06 9.75 8.90Cones Trilateral ASW 2.49 8.32 7.02
Bilateral ASW 3.97 9.79 8.26
27 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
ASW in Practice
28 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
GPU Adaptive Support Weight
Preliminary GPU ASWRuns as fast as 94 ms
(Quadro K6000)
ASW (from paper)[5]Takes almost 60 seconds
(CPU-only).
29 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
OpenCV Options
Block MatchingSemi-Global Block
MatchingDisparity Bilateral
Filter
Belief Propagation and Constant-Space Belief Propagation not shown
30 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Nvidia VisionWorks
Block Matching Semi-Global Block Matching
31 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Timing ComparisonTests ran on 450x375 “cones” images
Algorithm Jetson TK1 Jetson TX1 i7 + Quadro K6000
CPU BM 29ms 29ms 12msGPU BM 18ms 9.5ms 2.2msGPU DBF 64ms 21ms 10msCV SGBM 870ms 990ms 99msVX BM 13ms 5.7ms 2.3msVX SGBM 65ms 42ms 5.1msGPU ASW 8,900ms 6,800ms 94msCPU ASW 190,000ms 200,000ms 73,000ms
Source code for test programs can be found at:https://github.com/rdbeethe/gtc_tests
32 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Conclusion
ASW Pros:
• Excellent edge accuracy
• High disparity detail
• Less noise
• Parallelizes well
ASW Cons:
• Much highercomputational complexity
• No completed open-sourceGPU version
My GPU-accelerated ASW implementation can be found at:https://github.com/rdbeethe/asw
33 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
References I
Dongming Chen, Mohsen Ardabilian, and Liming Chen.A novel trilateral filter based adaptive support weightmethod for stereo matching.IEEE Transactions on Circuits and Systems for VideoTechnology, 25:730–743, 2014.
Asmaa Hosni, Michael Bleyer, and Margrit Gelautz.Secrets of adaptive support-weight for local stereomatching.Computer Vision and Image Understanding, 117:620–632,2013.
Daniel Scharstein and Richard Szeliski.A taxonomy and evaluation of dense two-frame stereocorrespondence algorithms.International Journal of Computer Vision, 2002.
34 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
References II
C. Tomasi and R. Manduchi.Bilateral filtering for gray and color images.IEEE International Conference on Computer Vision, 1998.
Kuk-Jin Yoon and In So Kweon.Adaptive support-weight approach for correspondencesearch.IEEE Transactions on Pattern Analysis and MachineIntelligence, 28:650–656, 2006.
35 / 36
ASW Stereo
Ryan Beethe
StereoMatching
Image Example
Block Matching
Matching Cost
BilateralFiltering
BilateralExplanation
AdaptiveSupport Weight
TrilateralFiltering
TrilateralExplanation
Trilateral-basedASW
ASW InPractice
GPU ASW
ASWAlternatives
TimingComparison
Conclusion
Questions?
36 / 36