shape-representation and shape similarity part 2 dr. rolf lakaemper
TRANSCRIPT
Shape-Representation
and
Shape SimilarityPART 2
Dr. Rolf Lakaemper
Motivation
WHY PARTS ?
Motivation
Motivation
Motivation
Motivation
Global similarity measures fail at:
• Occlusion• Global Deformation• Partial Match• (actually everything that occurs under ‘real’ conditions)
Parts
Requirements for a Part Based Shape Representation
(Siddiqi / Kimia ’96: ‘Parts of Visual Form: Computational Aspects’)
Parts
How should parts be defined / computed ?
Some approaches:
• Decomposition of interior• Skeletons• Maximally convex parts• Best combination of primitives
• Boundary Based• High Curvature Points• Constant Curvature Segments
Parts
Principal approach:
Hoffman/Richards (’85):
‘Part decomposition should precede part description’
=> No primitives, but general principles
Parts
No primitives, but general principals
“When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle)
Parts
“When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle)
Divide a plane curve into parts at negative minima of curvature
Parts
Different notions of parts:
• Parts: object is composed of rigid parts
• Protrusions: object arises from object by deformation due to a (growth) process (morphology)
• Bends: Parts are result of bending the base object
Parts
The Shape Triangle
Parts
This lecture focuses on parts, i.e. on partitioning a shape
Framework
A Framework for a Partitioning Scheme
Scheme must be invariant to 2 classes of changes:
• Global changes : translations, rotations & scaling of 2D shape, viewpoint,…
• Local changes: occlusions, movement of parts (rigid/non-rigid deformation)
Framework
A general decomposition of a shape should be based on the
interaction between two parts rather than on their shapes.
-> Partitioning by Part Lines
Framework
Definition 1:
A part line is a curve whose end points rest on the boundary of the shape, which is entirely embedded in it, and which divides it into two connected components.
Definition 2:
A partitioning scheme is a mapping of a connected region in the image to a finite set of connected regions separated by part-lines.
Framework
Definition 3:
A partitioning scheme is invariant if the part lines of a shape that is transformed by a combination of translations, rotations and scalings are transformed in exactly the same manner.
Framework
Definition 4:
A partitioning scheme is robust if for any two shapes A and B, which are exactly the same in some neighborhood N, the part lines contained in N for A and B are exactly equivalent.
Framework
Definition 5:
A partitioning scheme is stable if slight deformations of the boundary of a shape cause only slight changes in its part lines
Framework
Definition 6:
A partitioning scheme is scale-tuned if when moving from coarse to fine scale, part lines are only added, not removed, leading to a hierarchy of parts.
Framework
A general purpose partitioning scheme that is consistent with
these requirements is the partitioning by
limbs and necks
Framework
Definition :
A limb is a part-line going through a pair of negative curvature minima with co-circular boundary tangents on (at least) one side of the part-line
Limbs and Necks
Motivation: co-circularity
Limbs and Necks
The decomposition of the right figure is no longer intuitive: absence of ‘good continuation’
Smooth continuation: an example for
form from function
• Shape of object is given by natural function
• Different parts having different functions show sharp changes in the 3d surface of the connection
• Projection to 2d yields high curvature points
Limbs and Necks
Examples of limb based parts
Limbs and Necks
Definition :A neck is a part-line which is also a
local minimum of the diameter of an inscribed circle
Limbs and Necks
Motivation for necks: form from function
• Natural requirements (e.g. space for articulation and economy of mass at the connection) lead to a narrowing of the joint between two parts
Limbs and Necks
The Limb and Neck partitioning scheme is consistent with the
previously defined requirements
• Invariance• Robustness• Stability• Scale tuning
Limbs and Necks
Examples:
Limbs and Necks
The scheme presented does NOT include a similarity measure !
Limbs and Necks
Part Respecting Similarity Measures
Algorithms
Curvature Scale Space(Mokhtarian/Abbasi/Kittler)
A similarity measure implicitely respecting parts
CSS
CSS
Creation of reflection-point based feature-vector which implicitely contains part – information
CSS
Properties:
• Boundary Based• Continous Model (!) • Computes Feature Vector
• compact representation of shape• Performs well !
CSS
The idea:
• Smooth (continous) boundary curve using convolution with an increasing gaussian kernel
• Use the runlength position of curvature zero-crossings on the boundary as index set for each kernel size, thereby creating the ‘Curvature Scale Space’
• The maxima of the CSS are used for shape representation
• Similarity of shape is defined by difference between the maxima of the CSS representation
CSS
• Smooth (continous) boundary curve using convolution with an increasing gaussian kernel
Boundary curve S:
S={(x(u),y(u) | u [0,1]}
• Each coordinate of S is convolved with a 1D Gaussion kernel of width d
• The resulting curve S(d) is smoother than S
CSS
Inflection points (curvature zero crossings)
• Use the runlength position of curvature zero-crossings on the boundary as index set for each kernel size, thereby creating the
‘Curvature Scale Space’
CSS
CSS
• The maxima of the CSS are used for shape representation
CSS
• Similarity of shape is defined by difference between the maxima of the CSS representation
CSS
• Similarity of shape is defined by difference between the maxima of the CSS representation
CSS
Some results (Database: 450 marine animals)
CSS
Some results (Database: 450 marine animals)
CSS
Problems of CSS:• Convex shapes don’t have inflection points• Different shapes can have identical CSS !
CSS
The main problem:
CSS is continous, the computer vision world is discrete.
How to measure curvature in discrete boundaries ?
Dominant Points
Local curvature = average curvature in ‘region of support’
To define regions of support, ‘dominant points’ are needed !
Dominant Points
Dominant Points(“Things should be expressed as simple as possible, but not simpler”,
A. Einstein)
Idea: given a discrete boundary S compute polygonal boundary S’ with minimum number of vertices which is
visually similar to S.
Dominant Points
Example Algorithms( 3 of billions…)
• Ramer• Line Fitting
• Discrete Curve Evolution
DCE
Discrete Curve Evolution(Latecki / Lakaemper ’99)
Idea:
Detect subset of visually significant points
Curve Evolution
Target: reduce data by elimination of irrelevant features, preserve relevant features
... noise reduction
... shape simplification:
Curve Evolution: Tangent Space
Transformation from image-space to tangent-space
Tangent Space: Properties
In tangent space...
... the height of a step shows the turn-angle
... monotonic increasing intervals represent convex arcs
... height-shifting corresponds to rotation
... the resulting curve can be interpreted as 1 – dimensional signal => idea: filter signal in tangent space (demo: 'fishapplet')
Curve Evolution: Step Compensation
(Nonlinear) filter: merging of 2 steps with area – difference F given by:
pq p + q
F
F
F
q
p
Curve Evolution: Step Compensation
Interpretation in image – space:
... Polygon – linearization
... removal of visual irrelevant vertices
p q
removed vertex
Curve Evolution: Step Compensation
Interpretation in image – space:
... Polygon – linearization
... removal of visual irrelevant vertices
next:Iterative SC
Curve Evolution: Iterative Step Compensation
Keep it simple: repeated step compensation !
Remark: there are of course some traps ...
(demo: EvoApplet)
Curve Evolution: Iterative Step Compensation
Remark: there are of course some traps:
Self intersection / Topology preservation
Stop parameter
Edge movement
The evolution...
... reduces the shape-complexity
... is robust to noise
... is invariant to translation, scaling and rotation
... preserves the position of important vertices
... extracts line segments
... is in accord with visual perception
... offers noise-reduction and shape abstraction
... is parameter free
Curve Evolution: Properties
... is translatable to higher dimensions
Curve Evolution: Properties
Robustness (demo: noiseApplet)
Curve Evolution: Properties
Preservation of position, no blurring !
Strong relation to digital lines and segments
Curve Evolution: Properties
Noise reduction as well as shape abstraction
Curve Evolution: Properties
Parameter free (?)
Curve Evolution: Properties
Extendable to higher dimensions
Curve Evolution: Properties
Extendable to higher dimensions
Curve Evolution: Properties
Extendable to higher dimensions
Curve Evolution: Properties
Extendable to higher dimensions
Curve Evolution: Properties
Result: The DCE creates a polygonal shape representation in different levels of granularity: Scale Space Curvature can be defined as the turning angle at the vertices Regions of support are defined by vertices Easy traceable Scale Space is created, since no points are relocated
Curve Evolution: Properties
Scale Space
Ordered set of representations on different information levels
The polygonal representation achieved by the DCE has a huge
advantage:
It allows easy boundary partitioning using convex / concave
parts (remember the limbs !)
(MATLAB Demo MatchingDemo)
Polygonal Representation
Some results of part line decomposition:
DCE
The ASR (Advanced Shape Recognition) Algorithm uses the boundary parts achieved by the
polygonal representation for a part based similarity measure !
(Note: this is NOT the area partitioning shown in the previous slide)
ASR
The ASR is used in the ISS Database
ASR / ISS
The Interface (JAVA – Applet)
The Sketchpad: Query by Shape
The First Guess: Different Shape - Classes
Selected shape defines query by shape – class
Result
Specification of different shape in shape – class
Result
Let's go for another shape...
...first guess...
...and final result
Query by Shape, Texture and Keyword
Result
Behind The Scenes of the ISS - Database:
Modern Techniques of ShapeRecognition and Database Retrieval
How does it work ?
The 2nd Step First: Shape Comparison
Developed by Hamburg University in cooperation withSiemens AG, Munich, for industrial applications in...
... robotics
... multimedia (MPEG – 7)
ISS implements the ASR (Advanced Shape Recognition) Algorithm
Reticent Proudness…
MPEG-7: ASR outperformes classical approaches !
Similarity test (70 basic shapes, 20 different deformations):
Wavelet Contour Heinrich Hertz Institute Berlin 67.67 %
Multilayer Eigenvector Hyundai 70.33 %
Curvature Scale Space Mitsubishi ITE-VIL 75.44 %
ASR Hamburg Univ./Siemens AG 76.45 %
DAG Ordered Trees Mitsubishi/Princeton University 60.00 %
Zernicke Moments Hanyang University 70.22 %
(Capitulation :-) IBM --.-- %
Wide range of applications...
... recognition of complex and arbitrary patterns
... invariance to basic transformations
... results which are in accord with human perception
... parameter-free operation
Requirements
Robust automatic recognition of arbitrary shaped objectswhich is in accord with human visual perception
Industrial requirements...
... robustness
... low processing time
... applicable to three main tasks of recognition
Wide range of applications...
... recognition of complex and arbitrary patterns
... invariance to basic transformations
... results which are in accord with human perception
... parameter-free operation
Requirements
Robust automatic recognition of arbitrary shaped objectswhich is in accord with human visual perception
Industrial requirements...
... robustness
... low processing time
Next:StrategyScaling (or resolution)
Rotation
Rigid / non-rigid deformation
... applicable to three main tasks of recognition
Wide range of applications...
... recognition of complex and arbitrary patterns
... results which are in accord with human perception
... applicable to three main tasks of recognition
... parameter-free operation
Requirements
Robust automatic recognition of arbitrary shaped objectswhich is in accord with human visual perception
... robustness
Industrial requirements...
... robustness
... low processing time
... invariance to basic transformations
... low processing time
Simple Recognition (yes / no)
Common Rating (best of ...)
Analytical Rating (best of, but...)
Different Approaches
... Correlation
Pattern Matching...
Geometrical description...
... Hough – Transformation
Feature – Vectors...
... (Zernicke - ) Moments
Based on Visual Parts...
... Mokhtarian
... ASR
Curvature Scale Space (Mokhtarian, Mitsubishi)
Creation of reflection-point based feature-vector which implicitely contains part – information
ASR: StrategyASR: Strategy
Source: 2D - Image
Arc – Matching
Contour – Segmentation
Contour Extraction
Object - Segmentation
Evolution
ASR: StrategyASR: Strategy
Arc – Matching
DCE
Contour – Segmentation
Contour Segmentation
Correspondence ?
Similarity of parts ?
Part Similarity
Similarity of parts ?
= Boundary Similarity Measure
= Similarity of polygons
Tangent Space
Transformation from image-space to tangent-space
Shape Comparison: Measure
Tangent space offers an intuitive measure:
T g0 sT g1s a2ds
12 max l g0, l g1max
l g0l g1
,l g1l g 0
Shape Comparison: Measure
Drawback: not adaptive to unequally distributed noise if used globally !
…but works for single parts
Shape Comparison: Contour Segmentation
Shape Comparison: Correspondence
Optimal arc-correspondence:
find one to many (many to one) correspondence, that
minimizes the arc-measure !
next:Corr. -example
Graph of Correspondence
a0 a1 a2 a3
b0 b1 b2 b3
a0
b0
a1
a2a3
b1
b2b3
Graph:
... edge represents correspondence
... node represents matched arcs
arc
correspondence
Shape Comparison: Correspondence
Example:
a0 a1 a2 a3
b0 b1 b2 b3
a0
b0
a1
a2a3
b1
b2b3
Shape Comparison: Correspondence
Result:Optimal correspondence is given by cheapest way
next:Corr. - Results
Correspondence: Results
(MATLAB Demo)
Correspondence: Results
Correspondence and arc-measure allow...
... the identification of visual parts as well as
... the identification of the entire object
... a robust recognition of defective parts
... a shape matching which is in accord with human perception
ASR Results
Correspondence and arc-measure meet the requirements stated by Kimia et al.
Discrete
Easy computable
ASR: Applications in Computer Vision
Robotics: Shape Screening(Movie: Robot2.avi)
• Straightforward Training Phase
• Recognition of Rough Differences
• Recognition of Differences in Detail
• Recognition of Parts
ASR: Applications in Computer Vision
Application 2:
View Invariant Human Activity Recognition
(Dr. Cen Rao and Mubarak Shah, School of Electrical Engineering and Computer Science, University of Central Florida)
Application: Human Activity Recognition
Human Action Defined by TrajectoryAction Recognition by Comparison of Trajectories
(Movie: Trajectories)
• Rao / Shah:• Extraction of ‘Dynamic Instants’ by Analysis of Spatiotemporal Curvature• Comparison of ‘Dynamic Instants’ (Sets of unconnected points !)
• ASR: •Simplification of Trajectories by Curve Evolution• Comparison of Trajectories
Application: Human Activity Recognition
Trajectory Simplification
Activity Recognition: Typical Set of Trajectories
Trajectories in Tangent Space
Trajectory Comparison by ASR: Results
Recognition of 3D Objects by Projection
Background: MPEG 7 uses fixed view anglesImprovement: Automatic Detection of Key Views
Automatic Detection of Key Views
(Pairwise) Comparison of Adjacent Views•Detects Appearance of Hidden Parts
Automatic Detection of Key Views
Result (work in progress):
The Main Application: Back to ISS
Task:Create Image Database
Problem:Response TimeComparison of 2 Shapes: 23ms on Pentium1Ghz
ISS contains 15,000 images:Response Time about 6 min.
Clustering not possible:ASR failed on measuring dissimilarities !
Solution:
Full search on entire database using a simplercomparison
Vantage Objects (Vleugels / Veltkamp, 2000) provide a simple comparison of n- dimensional vectors (n typically < 100)
Vantage Objects
The Idea:Compare the query-shape q to a predefined
subset S of the shapes in the database D
The result is an n-dimensional Vantage Vector V,n = |S|
Vantage Objects
q
s1
s2
s3
sn
…
v1
v2
v3
vn
-- Each shape can be represented by a single Vantage Vector
-- The computation of the Vantage Vector calls theASR – comparison only n times
-- ISS uses 54 Vantage Objects, reducing the comparison time (needed to create the Vantage Vector) to < 1.5s
-- How to compare the query object to the database ?
Vantage Objects
-- Create the Vantage Vector vi for every shape di in the database D
-- Create the Vantage Vector vq for the query-shape q
-- compute the (euclidean) distance between vq and vi
-- best response is minimum distance
-Note: computing the Vantage Vectors for the database objects is an offline process !
Vantage Objects
-How to define the set S of Vantage Objects ?
Vantage Objects
-Algorithm 1 (Vleugels / Veltkamp 2000):
-Predefine the number n of Vantage Objects-S0 = { }-Iteratively add shapes di D\Si-1 to Si-1 such that
-Si = Si-1 di
-andk=1..i-1 e(di , sk) maximal. (e = eucl. dist.)
Stop if i = n.
Vantage Objects
-Result:
-Did not work for ISS.
Vantage Objects
-Algorithm 2 (Latecki / Henning / Lakaemper):
Def.: • A(s1,s2): ASR distance of shapes s1,s2
• q: query shape• ‘Vantage Query’ : determining the result r by minimizing e(vq , vi ) vi = Vantage Vector to si
• ‘ASR Query’: determining the result r by minimizing A(q,di )
Vantage Query has certain loss of retrieval quality compared to ASR query.
-Define a loss function l to model the extent of retrieval performance
Vantage Objects
Given a Database D and a set V of Vantage Vectors, the loss of retrieval performance for a single query by shape q is given by:
lV,D (q) = A(q,r),
Where r denotes the resulting shape of the vantage query to D using q.
Property:lV,D (q) is minimal if r is the result of the ASR-Query.
Vantage Objects
Now define retrieval error function L(S) of set S={s1 ,…, sn } D of Vantage Vectors of Database D:
L(S) = 1/n lS,D\{si} (si)
Task:Find subset S D such that L(S) is minimal.
Vantage Objects
Algorithm:
V0={ }iteratively determine sj in D\Sj-1 such that Sj =Sj-1 sj and L(Vj) minimal.
Stop if improvement is low
Vantage Objects
Result:Worked fine for ISS, though handpicked objects stil performed better.
Vantage Objects
HandpickedAlgorithm 2
Number of Vantage Objects
L(S)
…some of the Vantage Objects used in ISS:
Vantage Objects