november 2002 scene recognition by inexact graph matching1/39 alio/euro workshop scene recognition...
Post on 19-Dec-2015
218 Views
Preview:
TRANSCRIPT
November 2002
Scene recognition by inexact graph matching1/39 ALIO/EURO Workshop
Scene recognition by inexact graph
matchingCelso C. Ribeiro *Claudia BoeresIsabelle Bloch
IV ALIO/EURO Workshop on Applied Combinatorial Optimization
Pucón, Chile, November 4-6, 2002
November 2002
Scene recognition by inexact graph matching2/39 ALIO/EURO Workshop
Summary
• Motivation and applications• Optimal graph matching formulation• Model: objective function and
constraints• Randomized construction algorithm• Local search• GRASP heuristic• Numerical results• Current and future work
November 2002
Scene recognition by inexact graph matching3/39 ALIO/EURO Workshop
Motivation and applications• Recognition and understanding of complex
scenes require a detailed description of …– … the objects involved– … the spatial relationships between the objects
• The diversity of the forms of the same object in different instantiations of a scene, and also the similarities of different objects in the same scene, make the relationships between objects of prime importance to help the recognition of objects with similar appearance.
November 2002
Scene recognition by inexact graph matching4/39 ALIO/EURO Workshop
Motivation and applications• Graph-based representations often used
for scene representation:– Nodes represent the objects in the scene.– Edges represent the relationships between
the objects.• Relevant information is extracted from the
scene and represented by relational attributed graphs.
• Model-based recognition: both the model and the scene are represented by graphs.
Graph matching problem
November 2002
Scene recognition by inexact graph matching5/39 ALIO/EURO Workshop
Motivation and applications• Matching: graph or subgraph
isomorphism– The assumption of a bijection between the
elements in two instantiations of the same scene is too strong for some problems.
– Usually, the model has a schematic aspect.– Construction of the image graph often relies
on segmentation techniques that mail fail in accurately representing the image in meaningful entities, therefore no isomorphism can be expected.
– Recognition can be better expressed as an inexact (non-bijective) graph matching problem.
November 2002
Scene recognition by inexact graph matching6/39 ALIO/EURO Workshop
Motivation and applications• Application:
– Recognition of brain structures from 3D magnetic resonance images previously processed by a segmentation model (ENST and hospital Saint Vincent de Paul, Paris)
– Model is an anatomical atlas: nodes represent the anatomical structures, while edges represent the spatial relationships between them.
– Inaccuracies: oversegmented image, unexpected objects found (e.g. pathologies), expected objects not found, deformations, imprecise attributes
November 2002
Scene recognition by inexact graph matching7/39 ALIO/EURO Workshop
Motivation and applications• Axial slices of a brain (each grey level
corresponds to a unique connected structure): middle dark structures (lateral ventricles) are much bigger in (b) than in (a), the white hyper-signal structure (tumor) in (b) does not appear in (a) or (c).
(a) Normal brain
(c) Brain atlas
(b) Pathological brain
November 2002
Scene recognition by inexact graph matching8/39 ALIO/EURO Workshop
Motivation and applications
• Other applications:– Aerial or satellite image interpretation
using a map– Face recognition– Character recognition
November 2002
Scene recognition by inexact graph matching9/39 ALIO/EURO Workshop
Optimal graph matching formulation
• Attributed graphs: one single vertex for each image region
• Typical attributes: adjacencies, distances, relative positions, grey levels
• Vertex similarities and edge similarities are computed from the values of the attributed graphs, relating each pair of vertices and each pair of edges (one from the model, the other from the image).
November 2002
Scene recognition by inexact graph matching10/39 ALIO/EURO Workshop
Optimal graph matching formulation
• Model: G1=(N1,E1)
• Oversegmented image: G2=(N2,E2), with |N2|>|N1|
• Assign one single node of N1 to each node of N2:
• Set of nodes associated with i N1:
1 21, if node is associated with node
0, otherwiseij
i N j Nx
2( ) : 1ijA i j N x
November 2002
Scene recognition by inexact graph matching11/39 ALIO/EURO Workshop
Optimal graph matching formulation
• Edges (a,b) E1 and (p,q) E2 are associated if node a N1 is associated with p N2 and node b N1 is associated with q N2.
November 2002
Scene recognition by inexact graph matching12/39 ALIO/EURO Workshop
Optimal graph matching formulation
• Search for the optimal graph matching which best represents the scene recognition is based on similarity values.
November 2002
Scene recognition by inexact graph matching13/39 ALIO/EURO Workshop
Optimal graph matching formulation
• Similarity matrices:sv(i,j ): vertex-similarity between vertices i N1 and j N2.se(u,v ): edge-similarity between edges u = (a,b ) E1 and v = (c,d ) E2.
• A good matching is a solution in which the vertex associations and the edge associations correspond to high similarity values.
November 2002
Scene recognition by inexact graph matching14/39 ALIO/EURO Workshop
Model: objective function
1 2 1 2
(1 )max ( ) ( ) ( )v ef x f x f x
N N E E
1 2
( ) 1 ( , )v vij
i N j N
f x x s i j
node-similarities contribution
1 2( , ) ( , )
( ) 1 max , , , (( , ),( , ))e eap aq bp bq
ab E pq E
f x x x x x s ab pq
edge-similaritiescontribution
November 2002
Scene recognition by inexact graph matching15/39 ALIO/EURO Workshop
Model: objective function
1 2 1 2
(1 )max ( ) ( ) ( )v ef x f x f x
N N E E
High similarity associations are privileged:
sv(i,j ) high xij = 1 maximizes 1 ( , )vijx s i j
1 2
( ) 1 ( , )v vij
i N j N
f x x s i jnode-similarities contribution:
Low similarity associations are penalized:
sv(i,j ) low xij = 0 maximizes 1 ( , )vijx s i j
November 2002
Scene recognition by inexact graph matching16/39 ALIO/EURO Workshop
Model: constraints• Each vertex of N2 has to be associated
with exactly one vertex of N1:– Image segmentation is performed by an
appropriate algorithm which preserves the boundaries.
– In consequence, it avoids situations in which one region of the segmented image is located in the frontier of two adjacent regions of the model and no undersegmentation occurs.
Constraint (1): For every j N2, there exists exactly one node i N1 such that xij = 1, i.e.
1( ) 1.A j
November 2002
Scene recognition by inexact graph matching17/39 ALIO/EURO Workshop
Model: constraints
• Vertices of N2 associated with the same vertex of N1 should be all connected among themselves: – An oversegmentation method can split
an object into several pieces, but it does not change their relative positions (connectivity constraint).
Constraint (2): For every i N1, the graph inducedby A (i ) in G2 = (N2,E2) is connected.
November 2002
Scene recognition by inexact graph matching18/39 ALIO/EURO Workshop
Model: constraints
• Pairs of vertices with null similarity cannot be associated:
Constraint (3): sv(i,j ) = 0 xij = 0.
• All objects in the model should appear in the image:
Constraint (4): For every i N1, there exists at least one node j N2 such that xij = 1, i.e.
( ) 1.A i
November 2002
Scene recognition by inexact graph matching19/39 ALIO/EURO Workshop
Randomized construction algorithm• MaxTrials: maximum number of attempts to
build a feasible solution.• Repeat for at most MaxTrials attempts:
1. Set A (i ) = for every i N1.
2. Randomly select a node j N2 and set N2N2 – {j }.
3. Randomly select a node i N1, until sv(i,j ) > 0 and the graph induced in G2 by A (i ) {j } is connected.
4. Set A (i ) A (i ) {j }. 5. If N2 ≠ go back to step 2 above.
6. If A (i ) ≠ i N1 and A -1(j ) ≠ j N2, then stop.
November 2002
Scene recognition by inexact graph matching20/39 ALIO/EURO Workshop
Randomized construction algorithm
• The algorithm stops and returns the first feasible solution found within at most MaxTrials attempts.
• The algorithm enforces feasibility, good solutions are not necessarily obtained.
• It may also be extended to return the best among the first MaxSolutions feasible solutions built.
• Each attempt has time complexity .
2 2
1 2O N N
November 2002
Scene recognition by inexact graph matching21/39 ALIO/EURO Workshop
Randomized construction algorithm
November 2002
Scene recognition by inexact graph matching22/39 ALIO/EURO Workshop
Test problems• 12 test problems
– Some randomly generated, others from real problems associated with medical images
– Two difficult instances with isolated nodes– Largest instance has |N1|=50 and |E1|=88, |N2|
=250 and |E2|=1681.
• Construction algorithm is very fast and very often finds a feasible solution in the first attempt.
• Instances with isolated nodes are harder, but even in these cases a feasible solution is built.
November 2002
Scene recognition by inexact graph matching23/39 ALIO/EURO Workshop
Test problems: I-6 (human brain)
Model:12 vertices42 edges
Image:95 vertices1434 edges
November 2002
Scene recognition by inexact graph matching24/39 ALIO/EURO Workshop
Model:14 vertices27 edges
Image:28 vertices63 edges
Test problems: I-7 (muscle)
November 2002
Scene recognition by inexact graph matching25/39 ALIO/EURO Workshop
Test problems: I-7 (muscle)
model
oversegmented image
November 2002
Scene recognition by inexact graph matching26/39 ALIO/EURO Workshop
Test problems: I-7 (feasible solutions)
First feasible
solution: 4
regions
correctly
recognized
Best of ten
first feasible
solutions: 5
nodes
correctly
recognized
Best of 100
first feasible
solutions: 12
regions
correctly
recognized
November 2002
Scene recognition by inexact graph matching27/39 ALIO/EURO Workshop
Local search
• Solutions built by the construction algorithm are not necessarily optimal.
• Local search algorithm successively replaces the current solution by a better one in a neighborhood of the first, terminating at a local optimum.
• First improving strategy: the current solution is replaced by the first neighbor whose cost function value improves that of the current solution.
November 2002
Scene recognition by inexact graph matching28/39 ALIO/EURO Workshop
Local search: neighborhoods
• Neighborhood A: all solutions that can be obtained from the current one by modifying A -1(j ) for some node j N2 (the new node associated with j has to enforce all feasibility constraints).
• Neighborhood B: all solutions that can be obtained from the current one by exchanging the nodes A -1(p) and A -
1(q ) of N1, currently associated with nodes p and q of N2.
November 2002
Scene recognition by inexact graph matching29/39 ALIO/EURO Workshop
Local search: neighborhoods
Neighborhood A: associate a new node (a) with node 2
Neighborhood B: exchange the nodes b and c associated with nodes 4 and 7
November 2002
Scene recognition by inexact graph matching30/39 ALIO/EURO Workshop
Local search• VND procedure using neighborhoods A
and B and a first improving strategy:1. Start with a solution built by the
construction algorithm.2. Move to the first improving solution in
neighborhood A of the current solution until a local optimum is found.
3. If the current solution is also locally optimal with respect to neighborhood B, then stop.
4. Otherwise, move to the first improving solution in neighborhood B and go back to step 2.
November 2002
Scene recognition by inexact graph matching31/39 ALIO/EURO Workshop
Local search
• Each local search iteration (neighborhood search and move) has time complexity .
• Local search improved the solution value for all test problems.
2
1 2O N N
November 2002
Scene recognition by inexact graph matching32/39 ALIO/EURO Workshop
Local search: I-7
Local search applied
to the best of the 100
first feasible solutions
built: 13 regions correctly
recognized (out of 28)
November 2002
Scene recognition by inexact graph matching33/39 ALIO/EURO Workshop
GRASP heuristic• GRASP: multistart metaheuristic• Each iteration has two phases:
construction and local search• MaxIterations: maximum number of
iterations• Repeat for MaxIterations iterations:
– Build a feasible solution using the randomized construction algorithm.
– Use the local search procedure to improve the solution built.
– Update the best solution found.
November 2002
Scene recognition by inexact graph matching34/39 ALIO/EURO Workshop
GRASP heuristic
• Good solutions obtained with a few hundred GRASP iterations.
• Improved solutions for all test problems
• Small computation times• Problem I-7, 20000 GRASP
iterations: 216 seconds on a Pentium II 450 MHz
November 2002
Scene recognition by inexact graph matching35/39 ALIO/EURO Workshop
GRASP heuristic• Solution values for α = 0.9:
construction algorithm with MaxTrials = 500, first improving local search strategy, and 200 GRASP iterations
Problem
|N1| |E1| |N2| |E2| Construction
Local searc
h
GRASP
I-6 12 42 95 1434 0.412 0.425 0.426
I-7 14 27 28 63 0.618 0.634 0.637
I-8 30 39 100 297 0.498 0.518 0.521
I-9 50 88 250 1681 0.505 0.518 0.519
November 2002
Scene recognition by inexact graph matching36/39 ALIO/EURO Workshop
GRASP heuristic: I-720000 GRASP iterations, with local search starting from the first feasible solution built in the construction phase: 26 regions correctly recognized (out of 28)
November 2002
Scene recognition by inexact graph matching37/39 ALIO/EURO Workshop
GRASP heuristic: 1000 iterations
Problem I-6
Problem I-7
November 2002
Scene recognition by inexact graph matching38/39 ALIO/EURO Workshop
Current and future current work
• Improved construction algorithm incorporating a greedy (randomized) insertion criterion
• Improvement of the GRASP heuristic using path-relinking
• Parallelization on a 64-node Linux cluster using the MPI library for communication
• Comparison with other approaches in the literature, in particular with genetic algorithms
• Application to the recognition of facial features
November 2002
Scene recognition by inexact graph matching39/39 ALIO/EURO Workshop
Slides and publications• Slides of this talk can be downloaded from:
http://www.inf.puc-rio/~celso/talks• Papers about the randomized construction
heuristic with local search for scene recognition (shortly) available at:
http://www.inf.puc-rio.br/~celso/publicacoes
(also available: other papers about GRASP, parallelization, and path-relinking)
top related