6/9/2010 - techniontosca.cs.technion.ac.il/book/handouts/technionee2010_feature.pdf · 6/9/2010 9...
TRANSCRIPT
6/9/2010
1
1Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Feature-based methodsand shape retrieval problems
© Alexander & Michael Bronstein, 2006-2009© Michael Bronstein, 2010tosca.cs.technion.ac.il/book
048921 Advanced topics in visionProcessing and Analysis of Geometric Shapes
EE Technion, Spring 2010
2Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Structure
Local
Feature descriptors
Global
Metric
3Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Combining local and global structures
BBK 2008; Keriven, Torstensen 2009; Dubrovina, Kimme l 2010; Wang, B 2010
Pair-wise stress (global) Point-wise stress (local)
Local structure can be geometric or photometric
4Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Photometric stress
Thorstenstein & Keriven 2009
5Numerical Geometry of Non-Rigid Shapes Diffusion Geometry
Heat kernels, encore
Brownian motion on X starting at point x, measurable set C
probability of the Brownian motion to be in C at time t
Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005
Heat kernel represents transition probability
6Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Intrinsic descriptors
Sun, Ovsjanikov & Guibas 2009
Multiscale local shape descriptor (Heat kernel signa ture)
can be interpreted as probability of Brownian motion to return to
the same point after time (represents “stability” of the point)
Time (scale)
6/9/2010
2
7Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Sun, Ovsjanikov & Guibas 2009for small t
Relation to curvature
8Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Heat kernel signature
Heat kernel signatures represented in RGB space
Sun, Ovsjanikov & Guibas 2009Ovsjanikov, BB & Guibas 2009
9Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Heat kernel descriptors
Invariant to isometric deformations Localized sensitivity
to topological noise
J. Sun, M. Ovsjanikov, L. Guibas, SGP 2009M. Ovsjanikov, BB, L. Guibas, 2009
10Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Scale invariance
Original shape Scaled by
HKS= HKS=
Not scale invariant!
11Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Scale-invariant heat kernel signature
B, Kokkinos CVPR 2010
Log scale-space
Scaling = shift and multiplicative
constant in HKS
log + d/d ττττ
Undo scaling
Fourier transformmagnitude
Undo shift
0 100 200 300-15
-10
-5
0
τ0 100 200 300
-0.04
-0.03
-0.02
-0.01
0
τ0 2 4 6 8 10 12 14 16 18 20
0
1
2
3
4
ω=2kπ/T
12Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Scale invariance
B, Kokkinos 2009
Heat Kernel Signature Scale-invariantHeat Kernel Signature
6/9/2010
3
13Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Bending invariance
B, Kokkinos CVPR 2010
Heat Kernel Signature Scale-invariantHeat Kernel Signature
14Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Bending invariance
Wang, B 2010
Geodesic+HKS Diffusion+HKS Commute+SI-HKS
15Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Topology invariance
Geodesic+HKS Diffusion+HKS
Wang, B 2010
16Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Scale invariance
Wang, B 2010
Geodesic+HKS Commute+SI-HKS
17Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Invariance
Geodesic metric
Rigid Inelastic Topology
Diffusion metric
Scale
Wang, B 2010
Commute-timemetric
Heat kernelsignature (HKS)
Scale-invariant HKS (SI-HKS)
18Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
6/9/2010
4
19Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Tagged shapes
Shapes withoutmetadata
Man, person, humanPersonText search
Content-based search
3D warehouse
20Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
?
Content-based search problems
Invariant shape retrievalShape classification
?
Semantic
Variability of shape
within category
Geometric
Variability of shape
under transformation
21Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Image vs shape retrieval
Illumination View Missing data
Deformation Topology Missing data
22Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Bags of words
Notre Dame de Paris is a Gothic cathedral in the fourthquarter of Paris, France. It was the first Gothicarchitecture cathedral, and its construction spannedthe Gothic period.
cons
truc
tion
arch
itect
ure
Italy
Fra
nce
cath
edra
lch
urch
basi
lica
Par
isR
ome
Got
hic
Rom
an
St. Peter’s basilica is the largest church in world,located in Rome, Italy. As a work of architecture, it isregarded as the best building of its age in Italy.
Notre Dame de Paris is a Gothic cathedral in the fourthquarter of Paris, France. It was the first Gothicarchitecture cathedral, and its construction spannedthe Gothic period.
St. Peter’s basilica is the largest church in world,located in Rome, Italy. As a work of architecture, it isregarded as the best building of its age in Italy.
23Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Bags of features
Visual vocabulary
Feature detector + descriptor
Invariant to changes of the image
Discriminative (tells different images apart)
24Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Advantages
� “Shape signature”
� Easy to store
� Easy to compare
� Partial similarity possible
6/9/2010
5
25Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Images vs shapes
Images Shapes
Many prominent features Few prominent features
Affine transforms, illumination,
occlusions, resolution
Non-rigid deformations, topology,
missing parts, triangulation
SIFT, SURF, MSER, DAISY, … Curvature, conformal factor,
local distance histograms
26Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
“ShapeGoogle”
Feature descriptor
Geometric words
Bag of words
Geometric expressions
Spatially-sensitive bag of features
“ ”
“ ”
27Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Geometric vocabulary
M. Ovsjanikov, BB, L. Guibas, 2009
28Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Bags of features
Geometric vocabulary
M. Ovsjanikov, BB, L. Guibas, 2009
Nearest neighbor in the descriptor space
29Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Bags of features
Geometric vocabulary
M. Ovsjanikov, BB, L. Guibas, 2009
Weighted distance to words
in the vocabulary
30Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Bags of features
Shape distance = distance between bags of features
M. Ovsjanikov, BB, L. Guibas, 2009
Statistics of different geometric words over the entire shape
6/9/2010
6
31Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Index in vocabulary1 64
M. Ovsjanikov, BB, L. Guibas, 2009
Bags of features
32Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval
Statistical weighting
Query q Database D
syzygy in astronomy means alignment ofthree bodies of the solar system along astraight or nearly straight line. a planet isin syzygy with the earth and sun when itis in opposition or conjunction. the moonis in syzygy with the earth and sun whenit is new or full.
syzygy in astronomy means alignment ofthree bodies of the solar system along astraight or nearly straight line. a planet isin syzygy with the earth and sun when itis in opposition or conjunction. the moonis in syzygy with the earth and sun whenit is new or full.
Sivic & Zisserman 2003BB, Carmon & Kimmel 2009
Frequent in document= important
in is
or
syzygy
Rare in database= discriminative
with
a
of
the
and when
33Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval
Statistical weighting
Query q Database D
Significance of a term t
Term frequency Inverse documentfrequency
Weight bags of features by tf-idf
Reduce the influence of non-important terms in dense descriptor
Sivic & Zisserman 2003BB, Carmon & Kimmel 2009
34Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Expressions
In math science, matrixdecomposition is afactorization of a matrixinto some canonicalform. Each type ofdecomposition is used ina particular problem.
In biological science,decomposition is theprocess of organisms tobreak down into simplerform of matter. Usually,decomposition occursafter death.
Matrix is a science fictionmovie released in 1999.Matrix refers to asimulated reality createdby machines in order tosubdue the humanpopulation.
mat
rix d
ecom
posi
tion
mat
rix fa
ctor
izat
ion
scie
nce
fictio
nca
noni
cal f
orm
In math science, matrixdecomposition is afactorization of a matrixinto some canonicalform. Each type ofdecomposition is used ina particular problem.
In biological science,decomposition is theprocess of organisms tobreak down into simplerform of matter. Usually,decomposition occursafter death.
Matrix is a science fictionmovie released in 1999.Matrix refers to asimulated reality createdby machines in order tosubdue the humanpopulation.
mat
rixde
com
posi
tion is a
the of in to by
scie
nce
form
In math science, matrixdecomposition is afactorization of a matrixinto some canonicalform. Each type ofdecomposition is used ina particular problem.
Matrix is a science fictionmovie released in 1999.Matrix refers to asimulated reality createdby machines in order tosubdue the humanpopulation.
M. Ovsjanikov, BB, L. Guibas, 2009
35Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Expressions
In math science, matrixdecomposition is afactorization of a matrixinto some canonicalform. Each type ofdecomposition is used ina particular problem.
mat
rixde
com
posi
tion is a
the of in to by
scie
nce
form
In particular matrix usedtype a some science,decomposition form afactorization of iscanonical. matrix mathdecomposition is in aEach problem. into of
mat
rix d
ecom
posi
tion
mat
rix fa
ctor
izat
ion
scie
nce
fictio
nca
noni
cal f
orm
M. Ovsjanikov, BB, L. Guibas, 2009
36Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Visual expressions
“Inquisitor King” Inquisitor, King “King Inquisitor”
Giuseppe Verdi, Don Carlo, Metropolitan Opera
6/9/2010
7
37Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Geometric expressions
M. Ovsjanikov, BB, L. Guibas, 2009
“Yellow Yellow”Yellow
No total order between points (only “far” and “near”)
Geometric expression = a pair of spatially close geometric words
38Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Spatially-sensitive bags of features
M. Ovsjanikov, BB, L. Guibas, 2009
is the probability
to find word at point and
word at point
Proximity between
points and
Distribution of pairs of geometric words
Shape distance
is the statistic of geometric expressions of the form
39Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
M. Ovsjanikov, BB, L. Guibas, 2009
Spatially-sensitive bags of features
40Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
SHREC 2010 dataset
41Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
SHREC 2010 datasetBB et al, 3DOR 2010
42Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
ShapeGoogle with HKS descriptor (mAP %)BB et al, 3DOR 2010
6/9/2010
8
43Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
ShapeGoogle with SI-HKS descriptor (mAP %)BB et al, 3DOR 2010
44Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Scale 0.7 Heat Kernel Signature
?
Scale-Invariant Heat Kernel Signature
Scale-invariant retrieval
Kokkinos, B 2009
45Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Scale 1.3 Heat Kernel Signature
Scale-Invariant Heat Kernel Signature
Kokkinos, B 2009
Scale-invariant retrieval
46Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Heat Kernel SignatureLocalscale
Scale-Invariant Heat Kernel Signature
Kokkinos, B 2009
Scale-invariant retrieval
47Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Structure
Local
Feature descriptors
Global
Metric
48Michael Bronstein Diffusion geometry for shape recognition
Beylkin & Niyogi 2003Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005Rustamov 2007
Laplacian embedding
Represent the shape using finite-dimensional Laplacian eigenmap
Ambiguities!
6/9/2010
9
49Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Osada, Funkhouser, Chazelle & Dobkin 2002Rustamov 2007
Global point signature (GPS) embedding
☺ Deformation- and scale-invariant
☺ No ambiguities related to eigenfunction permutations and sign
☺ No need to compare multidimensional embeddings
Represent the shape using distribution of Euclidean distances in the
Laplacian embedding space (=commute time distances)
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45
50Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Diffusion distance distributions
Mahmoudi & Sapiro 2009
Represent the shape using distribution of diffusion distances
☺ Deformation-invariant � How to select the scale?
0.5 1 1.5 2 2.5 3 3.5 x 10-3
51Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Spectral shape distance
Kernel Distance Distribution Dissimilarity
Aggregation
BB 2010
52Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Spectral shape distance
Kernel Distance Distribution DissimilarityAggregation
BB 2010
53Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Spectral shape distance
Kernel Distance Distribution DissimilarityAggregation
Diffusion distance
BB 2010
54Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Particular case I: Rustamov GPS embedding
Kernel Distance Distribution DissimilarityAggregation
BB 2010
6/9/2010
10
55Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems
Particular case II: Mahmoudi&Sapiro
Kernel Distance Distribution DissimilarityAggregation
BB 2010