sketch-based 3d shape retrieval
DESCRIPTION
Mathias Eitz , Kristian Hildebrand, Tamy Boubekeur and Marc Alexa. Sketch-based 3D shape retrieval. Goal: sketch-based shape retrieval. input sketch. query result. query. retrieve. 3D model database. Outline. Background Sketches as input Overview Framework Results. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/1.jpg)
![Page 2: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/2.jpg)
SKETCH-BASED 3D SHAPE RETRIEVAL
Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur and Marc Alexa
![Page 3: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/3.jpg)
Goal: sketch-based shape retrieval
input sketch 3D model database
query
query result
retrieve
![Page 4: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/4.jpg)
![Page 5: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/5.jpg)
Outline
• Background– Sketches as input
• Overview• Framework• Results
![Page 6: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/6.jpg)
Why sketches as input?
• 3 common strategies for input
sketch (2D)
model (3D)
quick, simple, semanticskeywords
no/incorrect tags
rich input leads to good results
example often not available
rather simple, independent of external data
requires drawing skills
![Page 7: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/7.jpg)
Why sketches as input?
• Shape parts index into human memory [Hoffman’97]• 80-90% of lines explained by known definitions [Cole’08]
computer
≈human sketch
![Page 8: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/8.jpg)
Outline
• Background• Overview
– Previous work– Comparison with our approach
• Framework• Results
![Page 9: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/9.jpg)
Overview• Current retrieval systems rely on common
scheme:
0.210.130.750.310.41…
0.17 ,
s
0.210.130.750.310.41…
0.17
0.230.150.780.290.40…
0.15 ,
s≈
![Page 10: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/10.jpg)
Previous work: global features[Löffler, 2000]
[Funkhouser’03]
[Pu’05]
[Hou’07]
[Shin’07]
[Napoleon’09]
0.210.130.750.310.41…
0.17
global descriptorglobal analysis
![Page 11: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/11.jpg)
Our approach: local features
• Independent local features allow for:– translation invariance– partial matching– standard search data structures
Bag-of-features [Sivic’03]
......
![Page 12: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/12.jpg)
Outline
• Background• Overview• Framework
– Offline indexing– Learning visual vocabulary– Online search
• Results
![Page 13: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/13.jpg)
Offline indexing
...
...
generate ...
...
render
![Page 14: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/14.jpg)
Offline indexingVisual
vocabulary
addextract
200700
quantize
...
...
render
for all 50 views separately
![Page 15: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/15.jpg)
Offline indexing: view generation
• Uniformly sample bounding sphere: 50 samples
...
...
![Page 16: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/16.jpg)
Offline indexing: NPR lines
– Occluding contours– Suggestive contours [DeCarlo’03]
![Page 17: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/17.jpg)
...
Offline indexing: sampling & features
• Sampling: 500 random samples on lines• Representation: should be concise & robust
– local image statistics
![Page 18: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/18.jpg)
Offline indexing: features
• No directionality information in gradients• Binned distribution invariant to small deformations
(1) Extract local region (2) estimate orientations (3) distribution of orientations
4x4 spatial, 8 radial bins
![Page 19: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/19.jpg)
.
.
.
Offline indexing: visual vocabulary• 20k images (sampled from 50 views each of 2k models)• 500 local features each
– Training set size: 10 million local features
20k
rand
om im
ages
training set
![Page 20: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/20.jpg)
Offline indexing: visual vocabulary
k-means
Cluster centers form “visual vocabulary”
, , , ,...id: 0 1 2 499
…
:
training set
k=500
![Page 21: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/21.jpg)
Offline indexing: quantization
• Quantization allows for– More compact representation– Grouping of perceptually similar features
Featureto be quantized
, , , ,...id: 0 1 2 499
visual word 499
represented by
![Page 22: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/22.jpg)
Offline indexing: representation
, , , ,...id: 0 1 2 499
...0 0 0 0...1 0 0 0...1 0 1 0...1 0 2 1...1 0 1 1...1 0 2 2
histogram of “visual words”
![Page 23: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/23.jpg)
Offline indexing: representation
... =
. . .
1
0
2
2
![Page 24: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/24.jpg)
Online searchVisual
vocabulary
addextract
200700
quantize
...
...
render
for all 50 views separately
query
![Page 25: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/25.jpg)
Online Search
sample 500 locations 500 feature vectors
quantize
2007300
3
![Page 26: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/26.jpg)
Online search
• Images as (sparse) histograms of visual words
word id
# words
![Page 27: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/27.jpg)
Online search
• Similarity as angle in high-dimensional space• Vectors sparse: use inverted index
s
, ,
2007300
3
2036000
2
![Page 28: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/28.jpg)
Outline
• Background• Overview• Framework• Results
– Images– Discussion
![Page 29: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/29.jpg)
Results
• Based on Princeton shape db (~2k models)– ~10ms for a search
![Page 30: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/30.jpg)
Results
![Page 31: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/31.jpg)
Results
![Page 32: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/32.jpg)
Failure cases
• NPR methods require high resolution meshes• Sketches from “real users” can be quite abstract
![Page 33: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/33.jpg)
Future work
• View generation– canonical, “salient” views – which provides best retrieval?
• Feature representation– multi-scale, rotation-invariance?
• Larger datasets than the PSB models– Method fast enough to handle millions of models– Will it remain effective?
![Page 34: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/34.jpg)
Thanks
• Acknowledgements– Princeton shape benchmark [Shilane’04]– RTSC tool by Doug DeCarlo, Szymon Rusinkiewicz– Cited authors for images from their papers
![Page 35: Sketch-based 3D shape retrieval](https://reader034.vdocument.in/reader034/viewer/2022051317/56816712550346895ddb7e1e/html5/thumbnails/35.jpg)