a search-classify approach for cluttered indoor scene understanding liangliang nan 1, ke xie 1,...
TRANSCRIPT
![Page 1: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/1.jpg)
A Search-Classify Approach for Cluttered Indoor Scene Understanding
Liangliang Nan1, Ke Xie1, Andrei Sharf2
1 SIAT, China
2 Ben Gurion University, Israel
![Page 2: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/2.jpg)
Digitalization of indoor scenes
Indoor scenes from Google 3D Warehouse
![Page 3: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/3.jpg)
Acquisition of indoor scenes
![Page 4: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/4.jpg)
Goal
• Scene understanding
![Page 5: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/5.jpg)
Challenges
• Clutter– Densely populated– Arbitrary arrangements
• Partial representation– Occlusions
• Complex geometry
![Page 6: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/6.jpg)
Classification & Segmentation
• Two interleaved problems– What are the objects?– Where are the objects?
• Chicken-egg problem– Classification needs segmentation– Segmentation needs a prior
![Page 7: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/7.jpg)
Our solution
• Search – Propagate / accumulate patches
• Classify– Query classifier to detect object
![Page 8: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/8.jpg)
Related Work
• Indoor scenes (This Session)– [Fisher et al. 2012] [Shao et al. 2012] [Kim et al. 2012]
• Semantic relationship– [Fisher et al. 2010, 2011]
• Recognition using depth + texture (RGB-D)– [Quigley et al.2009], [Lai and Fox 2010]
• Outdoor classification– [Golovinskiy et al. 2009]
• Semantic labeling– [Koppula et al. 2011]
Controlled region growing process
![Page 9: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/9.jpg)
Our search-classify idea
0.6 0.8
0.92 0.94 0.94 0.94
![Page 10: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/10.jpg)
Method overview
Training
Search-Classify
![Page 11: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/11.jpg)
Point cloud features
– Height-size ratio of BBox– Aspect ratio of each layer– Bottom-top, mid-top size ratio– Change in COM along horizontal slabs
Bh
BdBw
![Page 12: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/12.jpg)
Classifier
• Handle missing data– Occlusion
• Random decision forest– Efficient multi-class classifier
• Trained with both scanned and synthetic data– Manually segmented and labeled– 510 chairs – 250 tables – 110 cabinets – 40 monitors etc.
[Shotton et al. 2008, 2011]
![Page 13: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/13.jpg)
Search-Classify
• Starts from seeds– Random patch triplets– Remove seeds with low confidence
• Accumulating neighbor patches– Highest classification confidence
• Stop condition– Steep decrease in classification confidence
0.65 0.92 0.93 0.88
Seed
![Page 14: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/14.jpg)
• Segmented - classified objects problems– Overlap, outliers, ambiguities etc.
• Refinement – Outliers = patches with large distance
Segmentation refinement by template fitting
![Page 15: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/15.jpg)
Template deformation
• Different styles for each class• Predefined scalable parts• Templates can deform [Xu et al. 2010]
![Page 16: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/16.jpg)
Template deformation
• Different styles for each class• Predefined scalable parts• Templates can deform [Xu et al. 2010]
![Page 17: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/17.jpg)
Fitting via template deformation
Confidence Fitting error Best fitting
• Best matching template– One-side Euclidean distance from points to template
![Page 18: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/18.jpg)
Results and discussion
![Page 19: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/19.jpg)
Results and discussion
![Page 20: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/20.jpg)
Results and discussion
• Scalability test with varied object density
0 (25) 1 (45) 5 (60)
![Page 21: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/21.jpg)
Results and discussion
• Comparison
Lai et al. 2011
Ours
![Page 22: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/22.jpg)
Limitation
• Upward assumption– Features– Template fitting
![Page 23: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/23.jpg)
Future work
• Contextual information
![Page 24: A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel](https://reader036.vdocument.in/reader036/viewer/2022062619/55189de9550346991f8b474e/html5/thumbnails/24.jpg)
Thank you