Using the Inner-Distance for Using the Inner-Distance for Classification of Articulated ShapesClassification of Articulated Shapes

Haibin Ling and David JacobsHaibin Ling and David Jacobs

Center for Automation ResearchUniversity of MarylandUniversity of Maryland

College ParkCollege Park

Problems: Three toysProblems: Three toys

ScheduleSchedule

1.1. Related workRelated work

2.2. The inner-distance and its propertiesThe inner-distance and its properties

3.3. Extension of shape context with the Extension of shape context with the inner-distanceinner-distance

4.4. Silhouette matching using dynamic Silhouette matching using dynamic programming based on the new programming based on the new descriporsdescripors

5.5. Experiment resultsExperiment results

Related WorksRelated Works Bending invariant signature for 3D surfaces, Bending invariant signature for 3D surfaces,

[Elad & Kimmel 2003]: geodesic distances + [Elad & Kimmel 2003]: geodesic distances + MDSMDS

Shape context, [Belongie et al. 2002]Shape context, [Belongie et al. 2002] Two categories of methods for handling part Two categories of methods for handling part

structures:structures: Supervised methods explicitly build models for part Supervised methods explicitly build models for part

structures through training. Grimson 1990, structures through training. Grimson 1990, Felzenszwalb and Huttenlocher 2003, Schneiderman Felzenszwalb and Huttenlocher 2003, Schneiderman and Kanade 2004, etc. and Kanade 2004, etc.

Unsupervised methods: do not depend on explicit part Unsupervised methods: do not depend on explicit part models. Basri et al., Siddiqi&etal, Sebastian et al. 04, models. Basri et al., Siddiqi&etal, Sebastian et al. 04, Gorelick et al., etc.Gorelick et al., etc.

The Inner DistanceThe Inner Distance

Given two points Given two points x, yx, y in a shape in a shape OO ( (O O is a is a connected and closed subset of Rconnected and closed subset of R22), the ), the iinner-distancenner-distance between between x,yx,y, denoted as , denoted as d(x,y;O)d(x,y;O), is defined as the length of the , is defined as the length of the shortest path connecting x and y shortest path connecting x and y withinwithin O. O.

Reduced to the Euclidean distance for Reduced to the Euclidean distance for convex object.convex object.

Affected by concavity of shapes – a hint of Affected by concavity of shapes – a hint of part structure.part structure.

Properties of the Inner-DistanceProperties of the Inner-Distance

Articulation InsensitivityArticulation Insensitivity Articulation invariant for ideal articulated shapesArticulation invariant for ideal articulated shapes For most pairs of points in an articulated shape, the For most pairs of points in an articulated shape, the

relative changes of their inner-distances are very relative changes of their inner-distances are very smallsmall

Capturing part structuresCapturing part structures Difficult to prove: the definition of part structure Difficult to prove: the definition of part structure

remains unclearremains unclear We show this with examples and analysis of We show this with examples and analysis of

experimental resultsexperimental results

A Model of Articulated ShapesA Model of Articulated Shapes

Oi is a part; Jij is a junction between Oi, Oj

Oi and Oj has no common points

diameter(Jij) < ε, The diameter is in the sense of the inner-distanceε is very small compared to the parts. When ε =0, all junctions degenerate to single

points, O is called an ideal articulated object.

Articulations between ShapesArticulations between Shapes

The articulation of shape O is a one-to-one The articulation of shape O is a one-to-one mapping mapping ff from O to O‘=f(O) from O to O‘=f(O)O' is also an articulated object, and O' is also an articulated object, and

decomposed to parts O’decomposed to parts O’I I and junctions J’and junctions J’ ijij

where O'where O'ii=f(O=f(Oii), J'), J'ijij=f(J=f(Jijij) . This preserves the ) . This preserves the

topology between the articulated parts.topology between the articulated parts. f is rigid (rotation and translation only) when f is rigid (rotation and translation only) when

limited on each part. This means inner-limited on each part. This means inner-distances distances within each partwithin each part will not change. will not change.

Examples of Articulated ShapesExamples of Articulated Shapes

NotationsNotations

f(P) f(P) denotes {f(x): x in P}denotes {f(x): x in P}C(xC(x11,x,x22;P);P) denotes a shortest path from x denotes a shortest path from x1 1

in P to xin P to x2 2 in P. (P is a closed and in P. (P is a closed and

connected subset of Rconnected subset of R22.. ‘‘ indicates the image of a point or a point indicates the image of a point or a point

set under articulation f. set under articulation f. [[ and and ]] denote the concatenation of paths. denote the concatenation of paths.

Changes of Inner-Distance within Changes of Inner-Distance within Parts and JunctionsParts and Junctions

Fact 1: the inner-distance within any part is Fact 1: the inner-distance within any part is invariant to articulationinvariant to articulation

Fact 2: for two points in a same junction, the change of Fact 2: for two points in a same junction, the change of the inner-distance under articulation is bounded by the inner-distance under articulation is bounded by εε

Articulation Insensitivity of the Articulation Insensitivity of the Inner-DistanceInner-Distance

Theorem: Let O be an articulated object and Theorem: Let O be an articulated object and f be an articulation of O as defined above. f be an articulation of O as defined above. Let x,y be two arbitrary points in O. Let x,y be two arbitrary points in O. Suppose the shortest path C(x,y;O) goes Suppose the shortest path C(x,y;O) goes through m different junctions in O and through m different junctions in O and C(x',y';O') goes through m' different C(x',y';O') goes through m' different junctions in O', thenjunctions in O', then

|d(x,y;O) - d(x',y';O')| < max{m,m'} |d(x,y;O) - d(x',y';O')| < max{m,m'} εε

Proof of the Articulation Insensitivity Proof of the Articulation Insensitivity of the Inner-Distanceof the Inner-Distance

1.1. Decompose the shortest path C(x,y;O) Decompose the shortest path C(x,y;O) into segments. Each segment is either into segments. Each segment is either within a part, or start and end in a same within a part, or start and end in a same junction.junction.

2.2. Construct a relaxed path in O’ according Construct a relaxed path in O’ according to the decomposition.to the decomposition.

3.3. Apply the two facts mentioned above.Apply the two facts mentioned above.

Illustration of the ProofIllustration of the Proof

(a) Decomposition of C(x,y;O) with (a) Decomposition of C(x,y;O) with x=px=p00,p,p11,p,p22,p,p33=y. Note that a segment can go =y. Note that a segment can go

through a junction more than once (e.g. pthrough a junction more than once (e.g. p11,p,p22).). (b) Construction of C”(x',y';O') in O'. Note that (b) Construction of C”(x',y';O') in O'. Note that

C”(x',y';O') is not the shortest path.C”(x',y';O') is not the shortest path.

The Inner-Distance and Part StructureThe Inner-Distance and Part Structure

The inner-distance captures part The inner-distance captures part structuresstructures

With the same sample points, the distributions of With the same sample points, the distributions of Euclidean distances between all pair of points are Euclidean distances between all pair of points are virtually indistinguishable for the four shapes, while the virtually indistinguishable for the four shapes, while the distributions of the inner-distances are quite differentdistributions of the inner-distances are quite different

Another Interesting CaseAnother Interesting Case

With about the same number of sample points, the four With about the same number of sample points, the four shapes are virtually indistinguishable using Euclidean shapes are virtually indistinguishable using Euclidean distances, while their distributions of the inner-distances distances, while their distributions of the inner-distances are quite different except for the first two shapes.are quite different except for the first two shapes.

1) None of the shapes has (explicit) parts. 1) None of the shapes has (explicit) parts. 2) More sample points will not affect much to the above 2) More sample points will not affect much to the above

statement.statement.

Computing the Inner-DistanceComputing the Inner-Distance

Using shortest path algorithmsUsing shortest path algorithmsBuild a graph on the sample points. For each Build a graph on the sample points. For each

pair of sample points x andy, if the line pair of sample points x andy, if the line segment connecting x and y falls entirely segment connecting x and y falls entirely within the object, then build an edge between within the object, then build an edge between them with the weight equal to the Euclidean them with the weight equal to the Euclidean distance |x-y|.distance |x-y|.

Apply a shortest path algorithm to the graph.Apply a shortest path algorithm to the graph.

Application of the Inner-distanceApplication of the Inner-distance

Extend the Extend the shape contextshape context [Belongie 2002] [Belongie 2002] for shape matching and comparisonfor shape matching and comparison

Dynamic programmingDynamic programming for silhouette for silhouette matchingmatching

Other ways:Other ways:Multidimensional ScalingMultidimensional Scaling (MDS), (MDS),

Elad&Kimmel 2003Elad&Kimmel 2003Shape distribution [Shape distribution [Osada et al. 2002]Osada et al. 2002]

Previous Work on Shape ContextPrevious Work on Shape Context

Given n sample points xGiven n sample points x11,x,x22,...,x,...,xnn on a shape, the on a shape, the

shape context at point xshape context at point x ii is defined as a histogram is defined as a histogram

hhii of the relative coordinates of the remaining n- of the relative coordinates of the remaining n-

1points [Belongie et al. 2002]:1points [Belongie et al. 2002]:

[Belongie et al. 2002] proposed combining the [Belongie et al. 2002] proposed combining the shape context and thin-plate-spline (SC+TPS) for shape context and thin-plate-spline (SC+TPS) for shape matching.shape matching.

Extension to Shape Context --Extension to Shape Context --Inner-distance Shape Context (IDSC)Inner-distance Shape Context (IDSC)

The Euclidean distance is directly replaced The Euclidean distance is directly replaced by the inner-distance.by the inner-distance.

The orientation between points is replaced The orientation between points is replaced by the by the inner-angleinner-angle, which is defined as the , which is defined as the angle between the tangential direction of angle between the tangential direction of the shortest path and the local tangential the shortest path and the local tangential on the shape boundary.on the shape boundary.

The Inner-AngleThe Inner-Angle

The inner-angle is insensitative to The inner-angle is insensitative to articulation.articulation.

Extension to Shape Context --Extension to Shape Context --ExamplesExamples

Silhouette Matching through Silhouette Matching through Dynamic ProgrammingDynamic Programming

Utilize the ordering provided by the Utilize the ordering provided by the contour.contour.

Fast and accurateFast and accurateOther works using dynamic programming: Other works using dynamic programming:

[Basri et al. 1998, Petrakis et al. 2002].[Basri et al. 1998, Petrakis et al. 2002].Bipartite matching: more general, less Bipartite matching: more general, less

constraint, slow.constraint, slow.

Experiment: Articulate DatasetExperiment: Articulate Dataset

Experiment: MPEG7 Shape-CE-1Experiment: MPEG7 Shape-CE-1

Analysis of Experiment: MPEG7Analysis of Experiment: MPEG7

Two retrieval examples for comparing SC and IDSC on the MPEG7 data set. The left column show two shapes to be retrieved: a beetle and an octopus. The four right rows show the top 1 to 9 matches, from top to bottom: SC and IDSC for the beetle, SC and IDSC for the octopus.

The Inner-Distance and Part StructureThe Inner-Distance and Part Structure

We observed this data set is difficult for We observed this data set is difficult for retrieval mainly due to the complex part retrieval mainly due to the complex part structures in the shapes, though they have structures in the shapes, though they have little articulation. This shows that the inner-little articulation. This shows that the inner-distance is effective at capturing part distance is effective at capturing part structures.structures.

The following experiments also The following experiments also demonstrate similar effect.demonstrate similar effect.

Experiment: Kimia Silhouette 1Experiment: Kimia Silhouette 1

Experiment: Kimia Silhouette 2Experiment: Kimia Silhouette 2

Experiment: Swedish Leaf DatasetExperiment: Swedish Leaf Dataset

Using 25 training samples and 50 testing samples per Using 25 training samples and 50 testing samples per species. Average correct ratios are:species. Average correct ratios are:

Combination of simple features: 82% [Soderkvist 2001]Combination of simple features: 82% [Soderkvist 2001] Fourier descriptors: 89.60%Fourier descriptors: 89.60% SC+DP: 88.12%SC+DP: 88.12% IDSC+DP: 94.13%IDSC+DP: 94.13%

Experiment: HExperiment: Huuman Body Matchingman Body Matching

Left: between adjacent frames. Right: silhouettes Left: between adjacent frames. Right: silhouettes separated by 20 frames. Only half of the matched pairs are separated by 20 frames. Only half of the matched pairs are shown for illustration.shown for illustration.