![Page 1: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/1.jpg)
Data-Driven Shape Analysis --- Shape Classification
Qi-xing Huang Stanford University
![Page 2: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/2.jpg)
Innovation in acquisition
[Zhou et al. 13] [Li et al. 13]
![Page 3: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/3.jpg)
Crowdsourcing
Warehouse
![Page 4: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/4.jpg)
Application
![Page 5: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/5.jpg)
Data-driven reconstruction [Shen el al. 12, Nan et al. 12]
![Page 6: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/6.jpg)
Data-driven modeling
[Funkhouser et al. 04]
[Schulz et al. 14]
[Xu et al. 12]
[Kalogerakis et al. 11,12]
![Page 7: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/7.jpg)
Data-driven shape segmentation
Best single shape segmentation [Chen et al. 09]
Joint shape segmentation [Huang et al. 11]
![Page 8: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/8.jpg)
Data-driven shape matching Blended intrinsic maps [Kim et al. 11]
Composite
Intermediate object
![Page 9: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/9.jpg)
![Page 10: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/10.jpg)
![Page 11: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/11.jpg)
![Page 12: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/12.jpg)
![Page 13: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/13.jpg)
Shape classification
![Page 14: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/14.jpg)
Shape classification tasks
Category level Fine-grained
lounge rocking
folding rex
![Page 15: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/15.jpg)
• Category level
– Shape comparison
• Fine-grained classification
• Future directions
Outline
![Page 16: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/16.jpg)
Category level
Dense labels
Relatively clean labels
![Page 17: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/17.jpg)
Similar shape voting
Chair
Chair
Chair
Stool
![Page 18: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/18.jpg)
Graph-based semi-supervised learning
Graph is the key!
![Page 19: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/19.jpg)
Result http://peter-pc.stanford.edu/ShapeNet/UI/
0.84
0.86
0.88
0.9
0.92
0.94
0.96
Accuracy
Recall
![Page 20: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/20.jpg)
• Represent each model by a shape descriptor.
• Compare shapes by comparing their shape descriptors
Shape comparison using descriptors
?
![Page 21: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/21.jpg)
• Support vector machines
– Semi-supervised support vector machines [Zhu and Goldenberg 09]
• Boosting techniques
– Joint boosting
• Deep learning [Hinton et al. 12]
Other techniques to consider
![Page 22: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/22.jpg)
• Shape distributions [Osada et al. 02]
• Spherical harmonics [Kazhdan et al. 03]
Shape descriptors
+ + = +
+ + +
Constant 1st Order 2nd Order 3rd Order
![Page 23: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/23.jpg)
• Light-field descriptor – State-of-the-art
– Need to align to factor out rotations
• Compare rendered images – Textures
– Curvatures
• Image descriptors – GIST
– HOG
Image based techniques
Not descriptors any more
![Page 24: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/24.jpg)
Shape classification tasks
Fine-grained Category level
lounge rocking
folding rex
![Page 25: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/25.jpg)
Fine-grained --- challenges
Sparse and noisy labels Features
![Page 26: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/26.jpg)
Aligning shapes
![Page 27: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/27.jpg)
Machine learning
Sparse/Noisy Labels
Graph based semi- supervised learning [Zhu 09]
Good graphs?
Distance metric learning
Side Windsor
![Page 28: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/28.jpg)
System overview
Input Shapes
1: With-arms
3: Windsor
2: Side
4: Rex
1,3 2,4 2,3 1,4
Shape Matching
Affine FFD
![Page 29: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/29.jpg)
System overview
Distance Learning
with-arms side windsor rex
Distance field Spin images
![Page 30: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/30.jpg)
System overview
Graph-Based Classification
with-arms side windsor rex
![Page 31: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/31.jpg)
System overview
Distance Learning
with-arms side windsor rex
Distance field Spin images
![Page 32: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/32.jpg)
Desired distance metrics
Global similarity of legs Local similarity of backs
Cantilever Windsor
![Page 33: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/33.jpg)
Linear distance metric model
Distance field Spin images Stretching ratio
where and what to compare
![Page 34: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/34.jpg)
Objective terms
Data term -- max-margin model
Bi-plane Straight Swept
Similar sets – must links : Mj
Dissimilar sets – cannot links : Dj
![Page 35: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/35.jpg)
Objective terms
Data term -- max-margin model
Bi-plane Straight Swept
Distance field Spin images Over-fitting
![Page 36: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/36.jpg)
Regularization Term I
Bi-plane Straight Swept Coefficients have to communicate -- within each class
Sparse
Distance field Spin images
![Page 37: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/37.jpg)
Regularization Term II
Coefficients have to communicate -- among the classes -- low rank [Amit et al 07]
Bi-plane Straight Swept
is low-rank
Trace-norm [Candes et al. 09]
![Page 38: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/38.jpg)
Convex optimization
Alternating direction method of multipliers [Boyd el al 11]:
![Page 39: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/39.jpg)
Low-rank Part-wise Agreement
Distance field
Spin images
![Page 40: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/40.jpg)
System overview
Graph-Based Classification
with-arms side windsor rex
![Page 41: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/41.jpg)
Multi-label classification
Swivel chairs
![Page 42: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/42.jpg)
Multi-label classification
Rocking chairs
![Page 43: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/43.jpg)
Multiple cuts – diffusion distances
Rocking chairs
![Page 44: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/44.jpg)
Joint classification via MRF
Candidate classifications -- states in the MRF model
Cut saliency Mutual correlation
Rocking Cantilever
![Page 45: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/45.jpg)
Benchmark evaluation
0
20
40
60
80
100
Plane Chair Car
Descriptor-Based
Per-class
Joint
Accuracy
![Page 46: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/46.jpg)
Benchmark evaluation
0
20
40
60
80
100
Plane Chair Car
Descriptor-Based
Per-class
Joint
Recall
![Page 47: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/47.jpg)
Shapenet (http://www.shapenet.org)
34 categories, 114 fine-grained classes, 76K shapes
With H. Su, Y. Li, and L. Guibas
![Page 48: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/48.jpg)
Comparison
![Page 49: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/49.jpg)
Comparison
![Page 50: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/50.jpg)
Comparison
![Page 51: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/51.jpg)
• Large-scale categorization --- Millions of shapes and thousands of categories
Future directions
![Page 52: Data-Driven Shape Analysis --- Shape Classificationgraphics.stanford.edu/courses/cs468-14-spring/... · System overview Input Shapes 1: With-arms 3: Windsor 2: Side 4: Rex 1,3 2,4](https://reader035.vdocument.in/reader035/viewer/2022070906/5f78681bb4ee494979061952/html5/thumbnails/52.jpg)
• Label propagation
Future directions
Floor lamp
Table glass lamp
Tiffany Table Lamp
Oil Lamp