Shape similarity and Visual PartsLongin Jan Latecki
Temple Univ., [email protected]
In cooperation withRolf Lakamper (Temple Univ.),
Dietrich Wolter (Univ. of Bremen)
2
Object Recognition Process:
Source:2D image of a 3D object
Matching: Correspondence of Visual Parts
Contour Segmentation
Contour Extraction
Object Segmentation
Evolution
3
What is this talk about ?Comparison of object shape based on object contours
Object contours are naturally obtained in Computer Vision, Robot Navigation, and other applications as polylines (polygonal curves).Shape similarity reduces to similarity of polylines.
Shape similarity of polylines is not so simple:•simple 1-1 vertex correspondence does not work•a scale problem
4
• only part of a polyline may be visible
5
Since polylines are obtained as boundary parts of objects in usually noisy sensor data (e.g., digital images):
1. two similar polylines do not need to have the same number of vertices, i.e., do not have to be of comparable level of detail,
2. do not have to be of comparable size,3. may have only a subpart that is similar and that has a
significant contribution to their shape = visual part
Cognitive Similarity Requirements
6
Comparable level of detail (1)
It can be achieved in the context of•a single object or•a group of objects (e.g., a query and a goal shape)
How can we achieve a comparable level of detail forall objects if we treat each object separately?
By placing each object on the same level of theScale Space hierarchy.
7
Comparable level of detail Scale Space
8
Discrete Curve Evolution (DCE) We achieve a comparable level of detail with DCE.
Before a similarity measure is applied, the shape of objects is simplified by DCE in order to
• reduce influence of noise,
• simplify the shape by removing irrelevant shape features without changing relevant shape features.
Discrete Curve Evolution (DCE)
u
v
w u
v
w
It yields a sequence: P=P0, ..., Pm
Pi+1 is obtained from Pi by deleting the vertices of Pi that have minimal relevance measure
K(v, Pi) = |d(u,v)+d(v,w)-d(u,w)|
>
10
Discrete Curve Evolution: Preservation of position, no blurring
11
Discrete Curve Evolution: robustness with respect to noise
12
Discrete Curve Evolution: extraction of linear segments
13
Discrete Curve Evolution: mathematical properties
Convexity Theorem (trivial)Discrete curve evolution (when applied to a polygon)
converges to a convex polygon. Continuity Theorem (nontrivial)Discrete curve evolution is continuous.
L. J. Latecki, R.-R. Ghadially, R. Lakämper, and U. Eckhardt: Continuity of the discrete curve evolution. Journal of Electronic Imaging 9, pp. 317-326, 2000.
Polygon Recovery (nontrivial)DCE allows to recover polygons from their digital images.L.J. Latecki and A. Rosenfeld: Recovering a Polygon form Noisy Data. Computer
Vision and Image Understanding (CVIU) 86, 1-20, 2002.
14
Comparable level of detail for DCE (=stop condition) is based on a threshold on the relevance measure
15
Comparable level of detail for DCE is based on a threshold on the relevance measure
16
Scale Space Approaches to Curve Evolution
1. reaction-diffusion PDEs
2. polygonal analogs of the PDE-evolution (Bruckstein et al. 1995)
3. approximation (e.g., Bengtsson and Eklundh 1991)
Main differences:
[to 1, 2:] Each vertex of the polygon is moved at a single evolution step, whereas in our approach the remaining vertices do not change their positions.
[to 1, 3:] Our approach is parameter-free(we only need a stop condition)
17
18
(3) Local similarity
We need local similarity measure,i.e., we need to compare polylines but not
polygons.
Global similarity measures fail at:• - partial visibility (occlusion)• - not uniformly distributed noise• - actually everything that occurs under• real conditions
19
Local similarity
A simple solution to local similarity measure:Given a target polylineT, find the most similar part P of polyline Q.
Consider all subpolylines of Q combinatorial explosion
We consider only all connected subsets P of Q, O(n²) in the number of vertices n of Q.
However, a connected subset P may be a distorted version of T.
20
Optimal Shape Similarity The optimal similarity of P to T is the similarity of
modified P to T, where modified P is P with all features that make P distinct from T removed.
},...,{ 1 nppP
PSSPTsS TTT :),(minarg*
PSSPTsPTos TT :),(min),(
Consider all subpolylines of P combinatorial explosion
s is a global similarity measure
21
T PT*
Q P
22
• All shape similarity measures presented in the literature are global measures.
• Although they can be applied to local parts, they are not optimal.
23
Presented approachA two-step solution:
1. We consider only all connected subsets P of Q, O(n²), n vertices of Q.
1. We use a greedy algorithm to compute optimal similarity, O(n²), n vertices of P.
We need a global similarity measure s.
24
Shape Similarity in Tangent Space
A polyline P is mapped to its turn angle function T(P)in the tangent space
the height of each step shows the turn-angle,
monotonically increasing intervals represent convex arcs,
height-shifting corresponds to rotation,
the resulting curve can be interpreted as 1D signal
25
Shape Similarity Measure: Arkin at al. PAMI 1991.
26
Shape Comparison: Measure
Drawback: not adaptive to unequally distributed noise
27
Shape Comparison: Contour Segmentation
Solution: use this measure only locally,i.e., apply only to corresponding parts:
28
Shape Similarity
29
30
Correspondence of visual parts: Results
31
Correspondence of visual parts: non-rigid deformation
L. J. Latecki and R. Lakämper: Shape Similarity Measure Based on Correspondence of Visual Parts. IEEE Trans. Pattern Analysis and Machine Intelligence 22, 2000.
L. J. Latecki and R. Lakämper: Application of Planar Shape Comparison to Object Retrieval in Image Databases. Pattern Recognition 35, 2002.
32
Comparison to deformation energy
33
Partial Shape Matching
This shape similarity measure works fine if the whole contouris given:• Great performance in the MPEG-7 competition• Life web-based shape search engine
Can it be applied when only contour parts are given?
Yes, but only in the context of optimal shape similarity.
34
Subpart SelectionWe can use sliding window on the contour of the object Qto find a given target part T.However, we cannot expect to cut part P of Q that exactlycorresponds to T, due to • imperfect size of the window • noise and change of view point• scale selection.
Therefore, a similarity measure must be able to overlook parts of Q whose shape is irrelevant w.r.t. Q.
T
Q
35
T PT*
Q P
36
Matching and SimplificationThe main idea of the proposed computation ofthe optimal similarity is shape matching and simplificationsimilar to DCE in one process.
The main idea:We recursively remove a vertices of P to obtain P’ such that P’is the most similar to T. os(T,P)= global minimum of s(T,P’).
We achieve a comparable level of detail (1)for polyline P in the context of target T.Observe that this approach also solves the stop conditionproblem (DCE is applied to each object separately).
37
T PT*
Q P
0 100 200 300400
450
500
550
600
650
0 100 200 300400
450
500
550
600
650
0 100 200 300400
450
500
550
600
650
0 100 200 300400
450
500
550
600
650
0 100 200 300400
450
500
550
600
650
0 100 200 300400
450
500
550
600
650
A B C
D E
optimal subset
F
PT*=E
38
0 1 2 5 7 8 10 15 20 22 25 300
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
A
B
C D
E
F
optimal subset
39
Matching and Simplification IIGiven a target T and a polyline P.
os(T,P) = global minimum S(T,P’) + S(P’,P),
where and are weights. The simplest weight assignment is = 1 and =0.
40
41
Target T
Most similar 21 shapes
optimal subsets
*
*
42
Applications•Shape-based object recognition•Retrieval in image and video databases•Object tracking
•Shape-based tracking of objects in laser scans•Robot mapping•Robot localization in top view maps
43
Shape-based tracking of objects in laser scans
An example top view image of LRF scan data
44
Shape-based tracking of objects in laser scans
Row scan data (left) is segmented into polyline (right)
45
Scan polylines simplification with DCE
46
Matching scans by shape
Since shape may be very simple,we also use cyclic order and proximity information
47
Shape-based tracking of objects in laser scansBuilding a global map
Demo movie
Why go beyond the simple proximity of points computedby minimizing the least squared distance of all points?
Robot may slip or turn (unconsciously):Shape and order of polylines remain similar even if the displacement is significant
48
Robot mapping and localization
Shape similarity useful for: • globally consistent mapping • localization when odometry is not given or unreliable
49
Future WorkLearning of visual parts:How to select the most distinctive parts of objects?Our approach is based on statistics and our shape similaritymeasures:We use statistical methods to find the smallest possible setof most different parts within a given class of objectsand the smallest possible set of most separating partsamong different object classes.
Previous approaches to find visual parts are based on differential geometry. They are static in that the parts will bealways the same for different classes of objects.Our approach is dynamic: selected parts depend on the objects seen.
50
Automatically Selected Parts