cs381v experiment presentationvision.cs.utexas.edu/381v-spring2016/slides/kuo-expt.pdf · the paper...
TRANSCRIPT
CS381V Experiment Presentation
Chun-Chen Kuo
The Paper• Indoor Segmentation and Support Inference from
RGBD Images. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. ECCV 2012.
50 100 150 200 250 300 350 400 450 500 550
50
100
150
200
250
300
350
400
Pipeline
segmentation support inference
Outline
• Run the segmentation pipeline
• Experiment on the segmentation pipeline
• Run the support inference pipeline
• Address strength and weakness
Outline
• Run the segmentation pipeline
• Experiment on the segmentation pipeline
• Run the support inference pipeline
• Address strength and weakness
Segmentation Pipeline
Image920, RGB Depth Map
Compute Surface Normal
y
x
z
Align to room coordinates
Aligned Surface Normal
y
x
z
After Alignment
Find Major Planes by RANSAC
0.2454x+0.1918y+0.9503z-4.2327
Reassign Pixels to Planes
Watershed Segmentation• Force the over-segmentation to be consistent with
the previous planes
1614
Hierarchical Grouping • Bottom-up grouping by boundary classifier
(Logistic regression AdaBoost)
309 145 85
78
77
AdaBoost Decision Tree
• Reweigh misclassified regions
• Optimize new tree with reweighed regions
• Score the tree
• Weighted sum over all trees optimized in each iteration
merge?
Final Regions
77
Ground truth
Outline
• Run the segmentation pipeline
• Experiment on the segmentation pipeline
• Run the support inference pipeline
• Address strength and weakness
Experiment on Segmentation Pipeline
• NYU Depth Dataset V2
• Images 909~1200
• Assign pixels to major planes
• AdaBoost decision tree as boundary classifier
Hypothesis• The trade-off between matching to 3D values,
normals, and gradient smoothing
• If alpha is small, neighbor pixels with similar RGB tend to be assigned to a same plane
• If alpha is large, match pixels to planes based on 3D points and normals, regardless gradient smoothing
Result of Plane Labeling
alpha=0
Result of Plane Labeling
alpha=2500
Result of Plane Labeling
alpha=0.25
Result of Plane Labeling
alpha=0.25alpha=0 alpha=2500
Segmentation Scorealpha=0.25 alpha=2.5alpha=0.25e-12
Hypothesis• Number of iteration of an AdaBoost decision forest
boundary classifier (underfit vs. overfit)
• At higher stage, the number of training example(boundary) decreases, causing lower accuracy and overfitting
• Accuracy at lower stage is more important because of error propagation
stage 1 stage 2 stage 3
stage 4 stage 5
ROC Curve at Stage 1• iteration = 30 • iteration = 5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.904903, Test AUC: 0.894981
trainingtesting
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.917977, Test AUC: 0.903215
trainingtesting
true
posi
tive
rate
false positive rate
ROC Curve at Stage 2• iteration = 30 • iteration = 5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.796447, Test AUC: 0.780641
trainingtesting
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.816867, Test AUC: 0.777968
trainingtesting
ROC Curve at Stage 3• iteration = 30 • iteration = 5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.762504, Test AUC: 0.746715
trainingtesting
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.802231, Test AUC: 0.737996
trainingtesting
ROC Curve at Stage 4• iteration = 30 • iteration = 5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.727054, Test AUC: 0.718036
trainingtesting
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.773312, Test AUC: 0.718135
trainingtesting
ROC Curve at Stage 5• iteration = 30 • iteration = 5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.727807, Test AUC: 0.720329
trainingtesting
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1Train AUC: 0.774677, Test AUC: 0.713322
trainingtesting
Accuracy versus Iteration at Stage 1
0 5 10 15 20 25 30iteration
0.83
0.84
0.85
0.86
0.87
0.88
0.89
0.9
0.91
0.92accuracy
trainingtesting
Accuracy versus Iteration at Stage 2
0 5 10 15 20 25 30iteration
0.77
0.78
0.79
0.8
0.81
0.82
0.83
0.84
0.85accuracy
trainingtesting
Accuracy versus Iteration at Stage 3
0 5 10 15 20 25 30iteration
0.73
0.74
0.75
0.76
0.77
0.78
0.79
0.8
0.81accuracy
trainingtesting
Accuracy versus Iteration at Stage 4
0 5 10 15 20 25 30iteration
0.71
0.72
0.73
0.74
0.75
0.76
0.77
0.78accuracy
trainingtesting
Accuracy versus Iteration at Stage 5
0 5 10 15 20 25 30iteration
0.71
0.72
0.73
0.74
0.75
0.76
0.77
0.78accuracy
trainingtesting
Segmentation Scoreiteration = [5 5 5 5 5]iteration = [10 10 10 10 10]iteration = [30 30 30 30 30]
Accuracy at lower stage is more important!
Segmentation Scoreiteration = [1 1 1 1 1]
Accuracy at lower stage is more important!
0 5 10 15 20 25 30iteration
0.83
0.84
0.85
0.86
0.87
0.88
0.89
0.9
0.91
0.92
accuracy
trainingtesting
Outline
• Run the segmentation pipeline
• Experiment on the segmentation pipeline
• Run the support inference pipeline
• Address strength and weakness
Support Inference Pipeline
Structure Class Classifier
Structure Class Classifier
Support Classifier
containment, geometry, and horz featuretake ~1 day to extract features for 292 images!
Support Classifier
Infer by Linear Program
6 minutes for an image!
Structure and Support Inference
stripe: incorrect structure prediction
groundfurniturepropsstructure
50 100 150 200 250 300 350 400 450 500 550
50
100
150
200
250
300
350
400
Structure and Support Inference
50 100 150 200 250 300 350 400 450 500 550
50
100
150
200
250
300
350
400
50 100 150 200 250 300 350 400 450 500 550
50
100
150
200
250
300
350
400
Structure and Support Inference
over-segmentation(color variance in an object)
clutter, small objectsout of 4 classes
Outline
• Run the segmentation pipeline
• Experiment on the segmentation pipeline
• Run the support inference pipeline
• Address strength and weakness
Strength• Reason joint assignment for structure and support
• ~73% accuracy if ground truth segmentation is given
Weakness• Slow in testing time
-5 minutes for feature extraction -6 minutes for inference by linear programming
• Clutters, small(thin) objects, color variance in objects
• Only 4 structure classes(no human, pet,…etc)
• ~55% accuracy if bottom up segmentation followed by support inference
Reference
• Code:http://cs.nyu.edu/~silberman/projects/indoor_scene_seg_sup.html