sensor fusion for semantic segmentation of urban...
TRANSCRIPT
Sensor Fusion for Semantic Segmentation of Urban Scenes
Richard Zhang1, Stefan A. Candra1, Kai Vetter12, Avideh Zakhor1
1University of California, Berkeley2Lawrence Berkeley National Laboratory
Presented at ICRA, May 2015
1
Goal
• Extract semantic information by fusing multiple modalities• Image
• LiDAR scan
• Challenges• Incorporate information from
multiple scales
• Fuse information from multiple modalities with non-overlapping sensor coverage
2
Top Level Pipeline
3
Previous WorkSegmentation Image Features Point cloud features Fusion
Sengupta et al.ICRA 2013
none Descriptive none majority votingof image
classifications in 3DHe and Upcroft.IROS 2013
Single segmentationimage only
Descriptive none
4
Previous WorkSegmentation Image Features Point cloud features Fusion
Sengupta et al.ICRA 2013
none Descriptive none majority votingof image
classifications in 3DHe and Upcroft.IROS 2013
Single segmentationimage only
Descriptive none
Cadena and Košecká. ICRA 2014.
Single segmentationimage only
Simple Simple Early fusion
5
Previous WorkSegmentation Image Features Point cloud features Fusion
Sengupta et al.ICRA 2013
none Descriptive none majority votingof image
classifications in 3DHe and Upcroft.IROS 2013
Single segmentationimage only
Descriptive none
Cadena and Košecká. ICRA 2014.
Single segmentationimage only
Simple Simple Early fusion
Oursmultiple segmentations
both domainsDescriptive Descriptive Late fusion
6
Segmentation
• Treating each pixel/point separately is computationally intensive
• Under-segmentation & Over-segmentation errors
Multiple segmentations performed to get superpixels/supervoxels
7
Segmentation
• Treating each pixel/point separately is computationally intensive
• Under-segmentation & Over-segmentation errors
Multiple segmentations performed to get superpixels/supervoxels
8
Feature Extraction
9
Point cloud supervoxel featuresImage superpixel features
Early FusionImage
features
Point cloudfeatures
Image-only segmentation
10
Imageinformation
only
Early FusionImage
features
Point cloudfeatures
Image-only segmentation
Point cloud-onlysegmentation
11
Imageinformation
only
Point cloudinformation
only
Early FusionImage
features
Point cloudfeatures
Overlappingsensor
coverage
Fused segmentation
Image-only segmentation
Point cloud-onlysegmentation
12
Imageinformation
only
Point cloudinformation
only
Early FusionImage
features
Point cloudfeatures
Overlappingsensor
coverage
Fused segmentation
Image-only segmentation
Point cloud-onlysegmentation
Large feature spaceLess training data
13
Imageinformation
only
Point cloudinformation
only
Late Fusion
Imagefeatures
Point cloudfeatures
Image-onlysegmentation
Overlappingsensor
coverage
Point cloud onlysegmentation
Fused segmentation
Image-only segmentation
Point cloud-onlysegmentation
14
Late Fusion
Imagefeatures
Point cloudfeatures
Image-onlysegmentation
Overlappingsensor
coverage
Point cloud onlysegmentation
Fused segmentation
Image-only segmentation
Point cloud-onlysegmentation
15
Mid-level
feature
Fusion
PC onlyPoint cloud unimodal segmentation
Ground Truth
16
Image unimodal segmentation
Fusion
PC only
Image unimodal segmentation
Ground Truth
Point cloud unimodal segmentation(projected onto image)
17
Fusion
Ground Truth
Late-fused multi-modal segmentation
18
PC only
Image unimodal segmentation
Point cloud unimodal segmentation(projected onto image)
PC only
Image unimodal segmentation
Point cloud unimodal segmentation(projected onto image)
Fusion
Ground Truth
Late-fused multi-modal segmentation
19
PC only
Image unimodal segmentation
Point cloud unimodal segmentation(projected onto image)
Fusion
Ground Truth
Late-fused multi-modal segmentation
20
Fusion
Ground Truth
Late-fused multi-modal segmentation
Fused segmentation & Post-processingConditional Random Field (CRF) 21
PC only
Image unimodal segmentation
Point cloud unimodal segmentation(projected onto image)
Fusion
22
PC only Image only FusedPerformance on overlapping region
Fusion
23
PC only Image only FusedPerformance on overlapping region
Fusion
24
PC only Image only FusedPerformance on overlapping region
Fusion
25
PC only Image only FusedPerformance on overlapping region
Results
• Quantitative Results• Train and test on KITTI1 dataset, augmented with
additional annotations
• 252 images across 8 sequences• 140 images for training, 112 for testing
1Geiger, et al. KITTI. CVPR 12 26
Results
• Quantitative Results• Train and test on KITTI1 dataset, augmented with
additional annotations
• 252 images across 8 sequences• 140 images for training, 112 for testing
1Geiger, et al. KITTI. CVPR 12 27
Results
• Quantitative Results• Train and test on KITTI1 dataset, augmented with
additional annotations
• 252 images across 8 sequences• 140 images for training, 112 for testing
1Geiger, et al. KITTI. CVPR 12 28
Results
• Quantitative Results• Train and test on KITTI1 dataset, augmented with
additional annotations
• 252 images across 8 sequences• 140 images for training, 112 for testing
1Geiger, et al. KITTI. CVPR 12 29
Thank you!
30