deep learning for stereo matching and related...

Deep Learning for stereo matching and related tasks

Matteo Poggi, 12 July 2017http://vision.disi.unibo.it/~mpoggi

http://vision.disi.unibo.it/~mpoggi


● Confidence measures for stereo matching○ Confidence measures based on machine and deep learning

○ Exploiting confidence measures

Overview

Confidence measures for stereoGiven a pair of rectified images, stereo aims at infer a disparity map encoding the horizontal displacement (i.e. disparity) between each pixel on the reference image and their matching pixel on the target image

To do so, stereo algorithms computes matching costs between each pixel on the reference image and their candidate corresponding pixels on target, building a cost volume of size WxHxD, being D the disparity range on which correspondences are searched

Confidence measures for stereoGiven a disparity map output of a stereoalgorithm, confidence measures assess thereliability of the disparities assigned to each pixel

Several confidence measures, processingdifferent cues from the stereo algorithm

● matching cost● cost curve● left-right consistency● distinctiveness

Confidence measures for stereoThe effectiveness of the different confidences is measures by ROC curves, in particular by comparingtheir AUC (Area Under Curve)

Pixels are sorted in decreasing order of confidence,the error rate is computed by sub-sampling thedisparity map according to that

Lower AUC, more effective confidence measure

Hu and Mordohai, A Quantitative Evaluation of Confidence Measures for Stereo Vision, TPAMI 2012

Confidence measures based on machine and deep learning State-of-the-art was represented by Park and Yoon (CVPR 2015), proposing a random forest classifier fed with multiple confidence measures and hand-crafted features from the disparity map, up to 22, to infer the correctness of disparity assignments.

They also use their novel confidence measure to improve the results of SGM and FCVF algorithms

Park and Yoon, Leveraging Stereo Matching with Learning-based Confidence Measures, CVPR 2015

The confidence measures used by Park and Yoon are extracted from the cost volume.

We proposed a novel set of hand-crafted features to infer confidence from the disparity map only, not requiring the cost volume. This make such approach possibly suitable for any custom 3D sensor

The hand-crafted features are extracted for each pixel and encode the number of neighboring pixels sharing the same disparity hypothesis, the amount of different hypothesis in the neighborhood, median, variance and deviation from median. All of them are computed at 5 different scales to obtain 20 features

Poggi and Mattoccia, Learning a general-purpose confidence measure based on O(1) features and a smarter aggregation strategy for semi global matching, 3DV 2016

Confidence measures based on machine and deep learning

The proposed measure was used to improve the results of SGM algorithm by reducing the streaking artifact affecting the single scanline optimization deployed by the algorithm

By detecting the streaking artifacts using the confidence measure, we weight the contribution from each scanline according to the predicted confidence



The proposed strategy deployed on a simpler version of SGM (4 scanlines) enables for more accurate disparity estimation with respect to more complex SGM implementations (8 scanlines)



Given the evidence that a reliable measure can be obtained from the disparity map only, we inquired about using deep learning to predict it.

We proposed a patch-based CNN architecture (CCNN) to infer the confidence for each pixel on a disparity map by processing the map itself

Poggi and Mattoccia, Learning from scratch a confidence measure, BMVC 2016


Results obtained with lightweight architecture (reduced number of connections between layers), running in a few seconds on CPU and 0.1 seconds on GPU. Latter tests with full connections between layers gave better results

Poggi and Mattoccia, Learning from scratch a confidence measure, BMVC 2016


The latest confidence measures based on machine learning and deep learning, as well as the availability of new datasets such as KITTIs and Middlebury v3 make the evaluation by Hu and Mordohai outdated

We carried out an extensive evaluation of 72 confidence measures using three stereo algorithms (AD-CENSUS, MC-CNN and SGM) on three datasets (KITTI 2012, KITTI 2015 and Middlebury v3). Among them, we consider 6 state-of-the-art confidence measures based on random forest and CNN, for those we evaluate their behavior with different amount of training data

Poggi, Tosi and Mattoccia, Quantitative evaluation of confidence measures in a machine learning world, ICCV 2017


Confidence measures have always been estimated pixel-wise, i.e. the confidence of neighboring pixels has never been taken into account

Guided by the local aggregation strategies for stereo matching costs, we deployed a convolutional neural network to locally “refine” the confidence maps obtained by well-known confidence measures

Poggi and Mattoccia, Learning to predict stereo reliability enforcing local consistency of confidence maps, CVPR 2017


Poggi and Mattoccia, Learning to predict stereo reliability enforcing local consistency of confidence maps, CVPR 2017


On the other hand, machine learning and deep learning are not always suitable, in particular when designing very constrained solutions (e.g., embedded devices, FPGAs, …)

While most of the custom stereo systems deploy simple confidence estimators such as left-right consistency or uniqueness constraint, more effective measures can be implemented with proper design strategies (e.g., fixed point arithmetic), negligibly reducing their effectiveness

Poggi, Tosi and Mattoccia, Efficient confidence measures for embedded stereo, ICIAP 2017


Exploiting confidence measures One of the main question concerning confidence measures is: how can they improve stereo?

Guided by the observation that different stereo algorithms are somehow complementary (Spyropoulos and Mordohai, Ensemble classifier for combining stereo matching, 3DV 2015)

We deployed a CCNN-like architecture to combine multiple disparity maps and obtain a more accurate, final result. The network takes as input the different disparity maps, compute confidence scores for each of them by casting a multi-label classification problem and choose, for each pixel, the disparity proposed by the most confident algorithm

Poggi and Mattoccia, Deep Stereo Fusion: combining multiple disparity hypotheses with deep-learning, 3DV 2016

Poggi and Mattoccia, Deep Stereo Fusion: combining multiple disparity hypotheses with deep-learning, 3DV 2016

Exploiting confidence measures

State-of-the-art confidence measures are very good at detecting outliers in a disparity map, thus can be used to filter them out and process a non-local refinement technique to obtain a more accurate result

Tosi, Poggi and Mattoccia, Learning to detect and take advantage of reliable anchor points for stereo refinement, under review 3DV 2017


Confidence can also be plugged into deep learning loss functions to train a deep stereo system in self-supervised fashion, using old-fashioned stereo algorithms rather than ground-truth labels

Tonioni, Poggi, Mattoccia and Di Stefano, Unsupervised adaptation for deep stereo, ICCV 2017


While non-learning based measures are less effective, they can be used to select a subset of few, very reliable points and of few, very wrong ones. By heuristically combining the subsets provided by different measures, we automatically generate confidence labels to be used for self-supervised training of the state-of-the-art machine learning based measures

Tosi, Poggi, Mattoccia, Tonioni and Di Stefano, Learning confidence measures in the wild, BMVC 2017


… questions?Thanks :)

Matteo Poggi, 12 July 2017http://vision.disi.unibo.it/~mpoggi



deep learning for stereo matching and related...

Documents