joint histogram based cost aggregation for stereo matching dongbo min, member, ieee, jiangbo lu,...
TRANSCRIPT
![Page 1: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/1.jpg)
Joint Histogram Based Cost Aggregation For Stereo
MatchingDongbo Min, Member, IEEE,
Jiangbo Lu, Member, IEEE,
Minh N. Do, Senior Member, IEEE
IEEE TRANSACTION ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2013
![Page 2: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/2.jpg)
Outline
• Introduction• Related Works• Proposed Method : Improve Cost Aggregation• Experimental Results• Conclusion
![Page 3: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/3.jpg)
Introduction
![Page 4: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/4.jpg)
Introduction• Goal: Perform efficient cost aggregation.• Solution : Joint histogram + reduce redundancy • Advantage : Low complexity but keep high-quality.
Cost InitializationCost AggregationRefinementOthers
≈70~75%
≈20~25%
≈5%
![Page 5: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/5.jpg)
Related Works
![Page 6: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/6.jpg)
Related Works• Complexity of aggregation: O(NBL)
• Reduce complexity approach• Scale image [8]• Bilateral filter [9,10]• Geodesic diffusion [11] • Guided filter [12] =>O(NL)
N : all pixels (W*H)B : window sizeL : disparity level
![Page 7: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/7.jpg)
Reference Paper• [8] D. Min and K. Sohn, “Cost aggregation and occlusion handling with WLS in
stereo matching,” IEEE Trans. on Image Processing, 2008.
• [9] C. Richardt, D. Orr, I. P. Davies, A. Criminisi, and N. A. Dodgson, “Real-time spatiotemporal stereo matching using the dual-cross- bilateral grid,” in European Conf. on Computer Vision, 2010
• [10] S. Paris and F. Durand, “A fast approximation of the bilateral filter using a signal processing approach,” International Journal of Computer Vision, 2009.
• [11] L. De-Maeztu, A. Villanueva, and R. Cabeza, “Near real-time stereo matching using geodesic diffusion,” IEEE Trans. Pattern Anal. Mach. Intell., 2012.
• [12] C.Rhemann,A.Hosni,M.Bleyer,C.Rother,andM.Gelautz,“Fast cost-volume filtering for visual correspondence and beyond,” in IEEE Conf. on Computer Vision and Pattern Recognition, 2011
![Page 8: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/8.jpg)
Proposed Method
![Page 9: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/9.jpg)
Local Method Algorithm• Cost initialization=>Truncated Absolute Difference
=>• Cost aggregation=>Weighted filter
• Disparity computation=>Winner take all
[4,8]
[4] K.-J. Yoon and I.-S. Kweon, “Adaptive support-weight approach for correspondence search,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 4, pp. 650–656, 2006. [8] D. Min and K. Sohn, “Cost aggregation and occlusion handling with WLS in stereo matching,” IEEE Trans. on Image Processing, vol. 17, no. 8, pp. 1431–1442, 2008.
![Page 10: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/10.jpg)
Improve Cost Aggregation• New formulation for aggregation• Remove normalization• Joint histogram representaion
• Compact representation for search range• Reduce disparity levels
• Spatial sampling of matching window• Regularly sampled neighboring pixels• Pixel-independent sampling
![Page 11: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/11.jpg)
New formulation for aggregation• Remove normalization
=>
• Joint histogram representaion
![Page 12: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/12.jpg)
Compact Search Range• Reason• The complexity of non-linear filtering is very high.• Lower cost values do NOT provide really influence.
• Solution• Choose the local maximum points.• Only select Dc(<<D) with descending order to be disparity candidates.
![Page 13: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/13.jpg)
Compact Search Range• Cost aggregation
=>
• MC(q): a subset of disparity levels whose size is Dc.
O( NBD )
O( NBDc )
N : all pixels (W*H)B : window sizeD : disparity level
![Page 14: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/14.jpg)
Dc = 60Final acc. = 93.7%
Compact Search Range• Non-occluded region of ‘Teddy’ image
Dc = 6Include GT = 91.8%Final acc. = 94.1%
Dc = 5 (Best)Final acc. = 94.2%
![Page 15: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/15.jpg)
Spatial Sampling of Matching Window• Reason• A large matching window and a well-defined weighting function leads to
high complexity.• Pixels should aggregate in the same object, NOT in the window.
• Solution• Color segmentation => time comsuming• Spatial sampling => easy but powerful
● ● ●
●●●●●
●
●●●
●●●●●
●
●●●
●●●●●
●
●●●
● ● ●
![Page 16: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/16.jpg)
Spatial Sampling of Matching Window• Cost aggregation
=>
• S : sampling ratio
O( NBDc )
O( NBDc / S2)
![Page 17: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/17.jpg)
Parameter definitionN : size of image B : size of matching window N(p)=W×WMD : disparity levels size=DMC : The subset of disparity size=DC<<DS : Sampling ratio
Pre-procseeing
![Page 18: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/18.jpg)
Experimental Results
![Page 19: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/19.jpg)
Experimental Results• Pre-processing• 5*5 Box filter
• Post-processing• Cross-checking technique• Weighted median filter (WMF)
• Device: Intel Xeon 2.8-GHz CPU (using a single core only) and a 6-GB RAM• Parameter setting
( ) = (1.5, 1.7, 31*31, 0.11, 13.5, 2.0)
![Page 20: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/20.jpg)
Experimental Results
(a) (b)
(c) (d)
![Page 21: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/21.jpg)
Experimental Results• Using too large box windows (7×7, 9×9) deteriorates the
quality, and incurs more computational overhead.
• Pre-filtering can be seen as the first cost aggregation step and serves the removal of noise.
![Page 22: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/22.jpg)
Experimental Results
Fig. 5. Performance evaluation: average percent (%) of bad matching pixels for ‘nonocc’, ‘all’ and ‘disc’ regions according to Dc and S.
2 better than 1
The smaller S, the better
![Page 23: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/23.jpg)
Experimental ResultsThe smaller S, the longer
The bigger Dc, the longer
![Page 24: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/24.jpg)
Experimental Results
• APBP : Average Percentage of Bad Pixels
![Page 25: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/25.jpg)
Ground truthError mapsResultsOriginal images
![Page 26: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/26.jpg)
Experimental Results
![Page 27: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/27.jpg)
Conclusion
![Page 28: Joint Histogram Based Cost Aggregation For Stereo Matching Dongbo Min, Member, IEEE, Jiangbo Lu, Member, IEEE, Minh N. Do, Senior Member, IEEE IEEE TRANSACTION](https://reader036.vdocument.in/reader036/viewer/2022062516/56649de45503460f94adad96/html5/thumbnails/28.jpg)
Conclusion• Contribution• Re-formulate the problem with the relaxed joint histogram.• Reduce the complexity of the joint histogram-based aggregation.• Achieved both accuracy and efficiency.
• Future work• More elaborate algorithms for selecting the subset of label hypotheses.• Estimate the optimal number Dc adaptively.• Extend the method to an optical flow estimation.