3d shape matching and retrieval - university of...

3D Shape Matching and Retrieval

Computer Animation and Visualisation

Lecture 17

Taku Komura

Overview

● Introduction– Motivation, applications

– Challenges, key issues

● Global approaches– Image-based Methods

– Graph-based Methods

● Local approaches– Curvature

● Bag of visual words– Heat Kernel Signature

Many 3D objects in the world today to shape match

● Medical, CAD/CAM design, 3D scans, animation,

CAD/CAM

Manufacturing and production (You and Tsai, 2010)

Synthesizing new objects using existing objects [Kalogerakis et al.]

Recognizing and registering scanned data

● Recognizing objects

● Searching in the database for similar data and registering

Biometrics

3D video sequences

● Characterize a motion (Huang et al., 2010)

3D retrieval for navigation (Goodall et al. 2004)

Archeology

● Matching for reconstruction (Huang et al. 2006)

Challenges● The geometry can be very different

– Rigid objects

– Articulated objects

– Deformable objects

● The topology can be very different ● The data can contain a lot of noise

– Topological / geometric noise

Geometric and Topological Noise

● Noise can be added when scanning or processing the data

geometric noise topological noise

Key issues ● Input format

– Point clouds, meshes, volume data

● What type of features/descriptors are used?● Are the features/descriptors translation/rotation/scale

invariant? ● Abstraction

– Representation

– Whether the method can handle objects with different geometry and topology

Matching problems

● Non-rigid matching

● Partial matching

● Transform a 3D object into a numeric/symbolic

representation– Feature vectors

– Graphs

● Compare two objects through their representationsThe global approachThe global approachThe global approachThe global approach

The Global Approach

Depth-buffer descriptor

● Image-based descriptor (Vranic 2004)– Pose normalization

– Depth-buffer construction

– Fourier transformations

– Selection of coefficients

Depth-buffer descriptor

● Pose normalization - Typical procedure– Translate the center of mass to the origin of the

coordinate system

● Rotate according to the largest spread● Scale to common size

Depth-buffer descriptor

● Pose normalization– Done using principal component

analysis

Depth-buffer descriptor

● Construction– Project the object into the faces of a bounding

rectangle

Fourier Transform

Transform into the Frequency Domain

Spherical Harmonics

● The Fourier transform for a spherical function

Other approaches

● Ray-based feature vector (Vranic 2004)

Issues: Requires Alignment

● Difficult to align objects – whose points are equally distributed

– symmetric around an axis

● A rotation invariant feature is preferred

Rotation Invariant Spherical Harmonic Representation

Kazhdan et al SGP03

● Given a spherical depth map, convert to spherical harmonics

● Generate a vector of the power of each frequency

Issues with image-based methods

● Not very suitable for articulated / deformable objects well – The depth information is not consistent when the

pose changes

Graph-based Approaches

● Computing a graph structure out of the geometry and compare objects by graph matching

● Two types of graphs introduced here – Medial axis

– Reeb graph

Medial Axis

● A set of points that have two or more closest points on the surface

● The trajectory of the maximal circles/spheres

Computing the Medial Axis[Amenta et al.], Powercrust

● Approximate Method: – Using the Voronoi diagram

– Sample points on the surface and then compute the Voronoi diagram

– A Voronoi vertex is a point with three (2D)/four(3D) or more closest sample points.

– The corresponding Voronoi ball is the unique ball centred at the vertex that contains the closest sample points in its boundary.

– For every sample point there is an inner pole, namely the farthest Voronoi vertex in the intersection of the shape and the Voronoi cell of the sample point.

– Connect the inner poles according to the adjacency of the samples to approximate the medial axis

Medial Axis: Noise filtering

● The medial axis is sensitive geometric noise (bumps at the surface)

● We need filter the noise to produce a medial axis that represents the shape well

Scale Axis Transform

● Removing the inner poles whose maximal spheres are absorbed by nearby larger maximal spheres when their sizes are scaled

https://www.youtube.com/watch?v=-VsFTXfFXHQ#t=82

Reeb Graph

● Named after Georges Reeb (A French mathematician)

● A graph structure that represents the object geometry

● Used a lot in computer graphics and pattern recognition

Reeb Graph

● Given a function that we call Morse function, that defines a scalar value on the surface of a manifold, the Reeb graph is a graph whose nodes represent the isolines that have the same values, and the edges represent the connection of the isolines between different values

Geodesic Distance

● The shortest distance over the surface

● An approximation can be computed using graph distance

● Geodesic distance can be used as the Morse function

Geodesic Integral[Hilaga et al. 2001]

● We can use the integral of the geodesic distance to resolve the ambiguity of setting the start point

● µ(v) = ∫p g(v,p) dS● Normalize

– µn(v) = (µ(v) – min(µ)) / min(µ)

Multiresolution Reeb Graph

● Graph matching can be costly so can use the multiresolution Reeb graph

● Binary discretization preserve parent-child relationships

● Exploit them for matching

[Hilaga et al. 2001]

Matching process

● Calculate similarity● Match nodes

– Find pairs with maximal similarity

– Preserve multires hierarchy topology

● Sum up similarity

[Hilaga et al. 2001]

Content-based Retrieval[Hilaga et al. 2001]

https://www.youtube.com/watch?v=xA0S_oEWD1Y

Summary for Graph-based Approaches

● Good for computing the similarity based on the articulated structures

● Rotation / scale invariant (if normalized) ● Medial axis is sensitive to geometric noise but

can do robust matching after filtering● Not robust to topological noise

geometric noise topological noise

Overview

● Introduction– Motivation, applications

– Challenges, key issues

● Global approaches– Image-based Methods

– Graph-based Methods

● Local approaches– Curvature

● Bag of visual words– Heat Kernel Signature

Local Features and Partial Matching

● Features based on local geometry ● Good for finding saliency ● Useful for partial shape matching● Use curvature

Curvature

● Amount by which a geometric object deviates from being flat, or straight

● The curvature is the magnitude of the rate of change of the tangent vector T:

● Or can be computed by

Segmenting and Matching based on Curvature

● Curvature has been used for segmenting 2D contours and matching

● Need multiscale analysis for robustness

(conduct smoothing, and compute the curvature at different levels)

Curvature on Surfaces : Principal Curvature (1)

● At a point p on the surface, we can define a normal plane that contains the normal vector

● It also contains a unique direction tangent to the surface and cut the surface in a plane curve

Curvature on Surfaces: Principal Curvature (2)

● The plane curve have different curvature for different normal planes

● The principal curvatures at p, denoted and , are the maximum and minimum values of this curvature.

● Special Curvature – Mean curvature :

– Gaussian curvature

Discrete Laplace-Beltrami [Meyer et al.]

Discrete Curvatures

Voronoi Region Area

● Sum the A below for all the triangles of 1-ring(v i)

If the triangle T is non-obtuse● A + ●

● If obtuse and vi is the obtuse angle,

A += area(T) / 2

– Else A += area (T) / 4

Extracting Saliency using Curvature

● The salient points can be extracted using the curvature

● Can use the curvature as an attribute at the surface for matching

Overview

● Introduction– Motivation, applications

– Challenges, key issues

● Global approaches– Image-based Methods

– Graph-based Methods

● Local approaches– Curvature

● Bag of visual words– Heat Kernel Signature

Heat Kernel Signature● A vector that represents the temperature at a

heated point as time passes

[t1, t2, t3, ...., tn]● The temperature after a short time describes

the local geometry● The temperature after a long time describes the

global geometry

Local Geometric Feature by HKS

● The heat in short time has a similar distribution to the curvature

Comparison of the shapes

● Using the heat kernel signature vector, conduct a K-means clustering and produce a geometric vocabulary P = {p1 , . . . , pV } of size V.

● For each point x, we can compute the feature distribution of the vocabulary

θ (x) = (θ1(x), . . . , θV(x)) where

Bag of Features

● Integrating the feature distribution over the entire shape X yields a V × 1 vector

● For comparing two shapes, we can compute the distance between bag of features

Example of Bag of Features

Summary: Heat Kernel Signature

● Can obtain both the local and global features● Robust against topological noise

– The heat does not diffuse much when there is a connection with less volume

● The bag of features provide a quick scheme to compare the objects

References● Kazhdan et al. “Rotation Invariant Spherical Harmonic Representation of 3D Shape

Descriptors”, SGP03

● Ivan Sipiran and Benjamin Bustos, “Shape Matching for 3D Retrieval and Recognition”, SIBGRAPI 2013 Tutorial , Arequipa - Perú, August 5, 2013

● ALEXANDER M. BRONSTEIN et al., Shape Google: Geometric Words and Expressions for Invariant Shape Retrieval, Transactions on Graphics 2011, 30(1)

● Mark Meyer et al. Discrete Differential-Geometry Operators for Triangulated 2-Manifolds

● Amenta et al. “The Power Curst”

● Joachim Giesen, Balint Miklos, Mark Pauly, Camille Wormser: The Scale Axis Transform, ACM Symposium on Computational Geometry 2009

●