structural pattern recognition - greyc
TRANSCRIPT
![Page 2: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/2.jpg)
Aims of Pattern Recognition
Design “Intelligent Systems”
I Intelligence ← inter legere (pick out, discern)I Torture by removal of inputsI One Def.:
The aggregate or global capacity of the individual to actpurposefully, to think rationally, and to deal effectively with
his environment.Wechsler, D (1944). The measurement of adult intelligence.Baltimore: Williams & Wilkins. ISBN 0-19-502296-3.
2 / 155
![Page 3: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/3.jpg)
Aims of Pattern Recognition
I Why is it challenging ?I A large part of our brain is devoted to the analysis of perceptions.I Computer Science will be present everywhere : Houses, Cars,
iTowns, Phones, laptop,. . .I Human interactions should overpass the screen/keyword
interactions.⇒ Computer should understand their environments and react
accordingly... and somehow become intelligent.
3 / 155
![Page 4: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/4.jpg)
Songs
I Recognition of simple commands:I Devices (phones, car, TV, house, fridge,. . . )I Standards of operators
I Identification of songs/Musics (Audio Fingerprint)I Copyright protection for You
Tube/Dalymotion. . .I New services for mobile devices
I Voice/Song representation:I A vector (set of Fourier coefficients,Wavelet transform,. . . )I A functionI A string
4 / 155
![Page 5: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/5.jpg)
Shapes
Confusion matrix
Classes
11
11
11
10 111
1 9 1
111 10
1 105 / 155
![Page 6: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/6.jpg)
Shapes
I Applications:I Character recognition,I Docking,I Identification of signatures,I Pose estimationI Detection & Characterization of
spines...
I Shape representations:I Vector of features (e.g. Legendre’s
moments),I String representation of the
boundaryI Graph representation of the
skeleton.
6 / 155
![Page 7: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/7.jpg)
Image/Video indexation
I Image Classification/Retrieval
All images related to a subject
(a) classic research engine, (b) [CVPR10]
I Objects Detection
Learn : object’s appearances and rela-tionships with context.
I Visual WordsIntermediate representationadapted to object representa-tions.
7 / 155
![Page 8: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/8.jpg)
Brain
I Decode brain activity.1. Submit stimuli to subjects,2. Measure brain activities,3. Retrieve stimuli from brain
activities.I Activity measurements:
1. Set of signals (for each active zone),2. Graph of zones with correlated
signals.
8 / 155
![Page 9: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/9.jpg)
Document Analysis
I Applications :I Structuring of document,I Retrieval and identification of
signatures,I Analysis of Technical Drawings,I Reading of addresses,I Analysis of old documents.
I Tools:I Graphs,I Shape descriptors,I Markov RF,
9 / 155
![Page 10: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/10.jpg)
Biometry
I People identification by:I Fingers ++,I Hands +,I Eyes +++,I Face +,I Voice,
I Features:I Set of points (minuties),I Measure of texture,. . .
10 / 155
![Page 11: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/11.jpg)
Video Analysis
I Aims :I Surveillance/tracking of people on a single/network of camera,I Detection of abnormal trajectories,
I Applications :I Security,I Sports,I iTowns
I Main tools:I Background subtraction,I histograms, vectors, graphs,strings. . .
11 / 155
![Page 12: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/12.jpg)
ChemoinformaticsBuilding a new molecule requires many attempts and many test:Time consuming, expensive.I Aims
I Predict physical/biological properties of molecules (VirtualScreening)
I Regression (Continuous properties),I Classification (Discrete Properties),
I Cancerous/not cancerous,I Active against some diseases (Aids, Depression,. . . ).
I ToolsI Vector of properties,I Molecular graph.
12 / 155
![Page 13: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/13.jpg)
Different Levels/Steps of Recognition
Level 0: Hand made classification (Expert Systems)If x>0.3 and y<1.5 then TUMORS
Level 1: Design of feature vectors/(di)similarity measures.Automatic Classification
Level 2: Automatic design of pertinent features / metric fromhuge amount of examples.
We will mainly study Level 1 systems within the structural patternrecognition framework.
13 / 155
![Page 14: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/14.jpg)
Statistical vs Structural Pattern Rec.
I Statistical pattern recognitionI Based on numerical descriptions of objects,I Focused on individual descriptions.
I Structural Pattern recognitionI Based on both numerical/symbolic description of objects,I Focused on the relationships between objects.
I Pros and Cons :I Statistical
I Many efficient algorithms exists to manipulate numerical values,I Individual description of objects may lead to poor descriptions,
I StructuralI Nicely describe both the individual objects and their relationships,I Based on complex structure (Graphs, Strings) which can not be
readily combined with numerical algorithms.
14 / 155
![Page 15: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/15.jpg)
Strings
I Let L be an alphabet (L : Rp, a sequence of symbols,combination of both,..)
I A finite string of length n is an element of Ln
I An infinite string is an element of L∞.I A circular string of Ln is a string such that s[n+ 1] = s[1].I Appear between any sequential relationships between objects:
I Temporal (trajectories, song, video,. . . ),I Spatial (DNA, text,shape’s boundaries. . . )
I Examples:I s=”Hello” : TextI s=”AGATACA”: DNAI s=”.2 .4 .04 1.0” Song fingerprint.
15 / 155
![Page 16: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/16.jpg)
String Edit distance
I Let s1 ans s2 denote two strings. The string edit distance(Levenshtein distance) is the minimum number of edits needed totransform s1 into s2, with the allowable edit operations beinginsertion, deletion, or substitution of a single character.
I Let s1 = ”restauration”, s2 = ”restaurant”I restauration→ restaurantion (insertion of n)I restaurantion→ restaurantio(removal of n )I restaurantio→ restaurant(removal of i and o )I d(“restauration”,”restaurant’)=4
I This edit distance is readily extended by associating differentcosts to each operation. In this case, the cost of a transformationis the sum of the costs of elementary operations and the editdistance is the transformation with minimal cost.
16 / 155
![Page 17: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/17.jpg)
Dynamic Programming
A computer science problem may be (efficiently) solved by dynamicprogramming if it has:Optimal substructure : The optimum we search for may be deduced
from the optimal solutions of sub-problems.Overlapping sub-problems : The naive recursive decomposition of a
problem into sub-problems leads to solve many timesthe same problem. Exemple: The Pascal triangle:(
np
)=(n− 1p− 1
)+(n− 1p
)(
107
)=(
96
)+(
97
)but(
96
)=(
85
)+(
86
)and(
97
)=(
86
)+(
87
).
In case of non overlapping problem the strategy iscalled divide and conquer.
17 / 155
![Page 18: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/18.jpg)
String Edit distance
I Dynamic programming :I Let us suppose that
I We want to compute the edit distance between s1 and s2 up toindexes i and j
I we know the optimal solution for any (k, l) such that k + l < i+ j.d(s1[1 . . . , i], s2[1 . . . , j]) =
Min
d(s1[1 . . . , i− 1], s2[1 . . . , j]) + csupp(s1[i]),d(s1[1 . . . , i], s2[1 . . . , j − 1]) + cadd(s2[j]),d(s1[1 . . . , i− 1], s2[1 . . . , j − 1]) + csub(si[i], s2[j])
I Ex:
d(toto, tata) = Min
d(tot, tata) + csupp(o)d(toto, tat) + cadd(a)d(tot, tat) + csub(o, a)
18 / 155
![Page 19: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/19.jpg)
String Edit distance
function LevenshteinDistance(char s1[1..m], char s2[1..n])d[0, 0]← 0for i = 1→ m do
d[i, 0]← d[i− 1, 0] + csupp(s1[i]) . Removal of s1 prefixesend forfor j = 1→ n do
d[0, j]← d[0, j − 1] + cadd(s2[j]) . Insertion of s2 prefixesend forfor j = 1→ n do
for i = 1→ m do
d[i, j]← min
d[i− 1, j] + csupp(s1[i]),d[i, j − 1] + cadd(s2[j]),d[i− 1, j − 1] + csub(s1[i], s2[j])
end for
end forreturn d[m,n]
end function
19 / 155
![Page 20: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/20.jpg)
String Edit distance
I Example :
S a t u r d a y0 1 2 3 4 5 6 7 8
S 1 0 1 2 3 4 5 6 7u 2 1 1 2 2 3 4 5 6n 3 2 2 2 3 3 4 5 6d 4 3 3 3 3 4 3 4 5a 5 4 3 4 4 4 4 3 4y 6 5 4 4 5 5 5 4 3
I Exercise:
I n g e g n e r i a0 1 2 3 4 5 6 7 8 9 10
E 1c 2o 3n 4o 5m 6i 7a 8
20 / 155
![Page 21: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/21.jpg)
Sting Edit distance and DTW
I Let x = (x1, . . . , xn) and y = (y1, . . . , ym) denote 2 strings.I An alignement π between x and y is defined by two vectors
(π1, π2) of length p < n+m− 1 such that:{1 ≤ π1(1) ≤ · · · ≤ π1(p) = n1 ≤ π2(1) ≤ · · · ≤ π2(p) = m
with unitary increasing and no repetitions:
∀i ∈ {1, . . . p−1}(π1(i+ 1)− π1(i)π2(i+ 1)− π2(i)
)∈{ (
01
)(10
)(11
)}I The Dynamic Time Warping (DTW) distance is then defined as:
DTW (x, y) = minπ∈A(n,m)
|p|∑i=1
csub(xπ1(i), yπ2(i))
where A(n,m) denotes the set of alignements between strings oflength n and m.
21 / 155
![Page 22: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/22.jpg)
Dynamic Time Warping
I Example:
r e s t a u r a t i o nr e s t a u r a n t
1 2 3 4 5 6 7 8 9 10 11 12 13π1 : 1 2 3 4 5 6 7 8 9 10 11 12 12π2 : 1 2 3 4 5 6 7 8 8 8 8 9 10
I Connection with edit Distance:
1 2 3 4 5 6 7 8 9 10 11 12 13π1 r e s t a u r a t i o n nπ2 r e s t a u r a a a a n t
22 / 155
![Page 23: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/23.jpg)
String Edit distance
I Complexity in O(nm)I Extensions :
I Circular string edit distance,I (Circular) string edit distance with rewritings,. . .
I Conclusion :I May satisfy all axioms of a distance (according tocsub, cadd, csupp),
I Efficient algorithm on small/usual pattern recognition tasks,
23 / 155
![Page 24: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/24.jpg)
Q grams
I Due to its complexity, string edit distance is a bit useless on verylong strings (songs, DNA,long texts,. . . ).
I → need to localize quickly promising parts of 2 strings with a lowedit distance or to bound (quickly) the edit distance between twostring
I Given s1[1 . . . , n] and s2[1 . . . ,m] (n >> m) find all sub strings sof s1 such that d(s, s2) ≤ k.
I A q gram is a word of length q.I The set of q grams of a string s may be found in |s| and stored in
a databased together with its location.I Q grams distance between strings :
Dq(x, y) =∑s∈Σq |Hs(x)−Hs(y)|
=∑s∈Gr(x)∩Gr(y) |Hs(x)−Hs(y)|+
∑s∈Gr(x)−Gr(y) |Hs(x)|
+∑s∈Gr(y)−Gr(x) |Hs(y|)
Gr(x): Set of q grams of x.24 / 155
![Page 25: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/25.jpg)
Q gram and edit distance
I Fundamental theorem : Let x be a pattern string and y a text(song, DNA,. . . ) :
d(x, y) < k ⇒ Gr(x) ∩Gr(y) ≥ |x| − q + 1− kq
I Given any part y of a text T . Gr(y) may be computed linearlyand if Gr(x) ∩Gr(y) < |x| − q + 1− kq then d(x, y) ≥ k and ymay be rejected without computing d(x, y).
25 / 155
![Page 26: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/26.jpg)
An application to audio FingerprintI Compute the fingerprints of a database of songs.
I Each fingerprint is a stringI One element of a string ≈ every 5/10ms.I Each fingerprint is a string of approx. 18,000 elements.I Several thousands of songs in the database.
I Let I be an input song of 5 seconds. For each q gram of I, let Sqdenote the set of fingerprints D of the database such thatq ∈ Gr(D).
scoreI,D(q) =p∑k=1
q∑l=1
S(I[ik, ik +m], D[jl, jl +m])
ik, jl: occurrences of q in I and D. m : small integer, S similaritymeasure.
I Score between I and D:score(I,D) =
∑q∈Gr(x)∩Gr(y)
scoreI,D(q)
I Let Dmax the database fingerprint with the maximal score, andimax, jmax the locations in I and Dmax providing the highest qgram score:
score(I) = score(I[imax, imax +M ], D[jmax, jmax +M ])where M >> m.
26 / 155
![Page 27: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/27.jpg)
Bipartitite Assignement
I Let us consider two sets S (Engineers) and P (Projects) of size nand a cost function from S × T to R+.
I c(i, j) is the cost of engineer i for project j.I The aim of bipartite assigment algorithm is to determine a
bijective mapping function ψ : S → T (a permutation) fromEngineers to Projects which minimizes:∑
i∈Sc(i, ψ(i))
ψ is an optimal assigment minimizing the mapping of Engineersto projects.
I This problem may be formalize as the one of determining amapping on a bipartite weighted graph G = (S ∪ T,E, ) whereE ⊂ S × T .
I This problem is solved in O(n3)
27 / 155
![Page 28: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/28.jpg)
Mathématical modelTo each permutation ψ we associate the permutation matrix:
xij ={
1 If ψ(i) = j0 otherwize
The assignment problem is then translated into the search of apermutation matrix x∗ such that:
x∗ = argminx
n∑i=1
n∑j=1
cijxij
such that:I Each line of x sums to 1:
∀i ∈ {1, . . . , n}n∑j=1
xij = 1
I Each column of x sums to 1:
∀j ∈ {1, . . . , n}n∑i=1
xij = 1
I x is composed of 0 and 1: ∀(i, j) ∈ {1, . . . , n}2xij ∈ {0, 1} 28 / 155
![Page 29: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/29.jpg)
Problem reformulation
Let A be the 2n× n2 matrix defined by:
x11 . . . x1n x21 . . . x2n . . . xn1 . . . xnn
1 1 1 . . . 11 1 . . . 1
. . .n 1 1 . . . 1
n+ 1 1 1 11 1 1. . . . . . . . . . . .
2n 1 1 1
Using c and x as vectors the problem becomes:
minxctx such that Ax = 1
Note that A is unimodular (the determinant of any squared submatrix is 0, +1, or -1). Since A is unimodular and 1 is a vector ofinteger, x is a permutation matrix.
29 / 155
![Page 30: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/30.jpg)
Dual problems
Our initial problem is equivalent to :
MinAx=1,x≥0
ctx⇔ MaxAty≤c
1ty
Let y =(uv
)denotes the vector of 2n variables. Our problem is
thus equivalent to:
Max
n∑i=1
ui +n∑j=1
vj with ui + vj ≤ cij∀(i, j) ∈ {1, . . . , n}2
Algorithms may be decomposed into:1. Primal methods2. Dual methods3. Primal/Dual methods
30 / 155
![Page 31: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/31.jpg)
A dual methodI Rem. Source : Wikipedia.I Let us call a function y : (S ∪ T )→ R a potential if:
yi + yj ≤ ci,j∀i ∈ S, j ∈ T.
I The value of potential y is∑i∈S∪T yi.
I The cost of each perfect matching is at least the value of eachpotential.
I The Hungarian method finds a perfect matching and a potentialwith equal cost/value which proves the optimality of both.
I An edge ij is called tight for a potential y if yi + yj = ci,j .I Let us denote the subgraph of tight edges by Gy.I The cost of a perfect matching in Gy (if there is one) equals the
value of y.I All edges in Gy are initially oriented from S to T .I Edges added to the matching M ⊂ Gy are oriented from T to S.
Initial value of M = ∅.I y is intialised to 0.
31 / 155
![Page 32: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/32.jpg)
A dual method
I We maintain the invariant that all the edges of M are tight. Weare done if M is a perfect matching.
I In a general step,I let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, then reverse the orientation of a directed path in−→Gy from RS to RT . Thus the size of the corresponding matchingincreases by 1.
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on the vertices of Z ∩ S and decrease y by ∆ onthe vertices of Z ∩ T . The resulting y is still a potential. Thegraph Gy changes, but it still contains M.
I We repeat these steps until M is a perfect matching, in whichcase it gives a minimum cost assignment.
32 / 155
![Page 33: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/33.jpg)
A dual method: Example
Defs.I Let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − y − i− y − j : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example1 2 3 4
a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
a
b
c
d
1
2
3
4
S
T
33 / 155
![Page 34: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/34.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 0
a(0)
b(0)
c(0)
d(0)
1(0)
2(0)
3(0)
4(0)
S
T Ey = ∅;RS = S;RT = T ;Z = S;∆ = 1
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 35: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/35.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 1
a(1)
b(1)
c(1)
d(1)
1(0)
2(0)
3(0)
4(0)
S
T
Ey = (c, 1);RS = S;RT = T ;Z = S ∪ {1};
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 36: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/36.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 2
a(1)
b(1)
c(1)
d(1)
1(0)
2(0)
3(0)
4(0)
S
T Ey = (1, c);RS = {a, b, d};RT = {2, 3, 4};Z = {a, b, d};∆ = 1
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 37: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/37.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 3
a(2)
b(2)
c(1)
d(2)
1(0)
2(0)
3(0)
4(0)
S
T
Ey = {(1, c), (b, 2)};RS = {a, b, d};RT = {2, 3, 4};Z = {a, b, d, 2};
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 38: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/38.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 4
a(2)
b(2)
c(1)
d(2)
1(0)
2(0)
3(0)
4(0)
S
T Ey = {(1, c), (2, b)};RS = {a, d};RT = {3, 4};Z = {a, d};∆ = 2
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 39: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/39.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 5
a(4)
b(2)
c(1)
d(4)
1(0)
2(0)
3(0)
4(0)
S
T Ey = {(1, c), (2, b)};RS = {a, d};RT = {3, 4};Z = {a, d, 1, c, 2, b};∆ = 1
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 40: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/40.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 6
a(5)
b(3)
c(2)
d(5)
1(-1)
2(-1)
3(0)
4(0)
S
T
Ey = {(1, c), (2, b), (d, 4)};RS = {a, d};RT = {3, 4};Z = {a, d, 1, c, 2, b, 4};
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 41: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/41.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 7
a(5)
b(3)
c(2)
d(5)
1(-1)
2(-1)
3(0)
4(0)
S
T RS = {a};RT = {3};Z = {a, 1, c};T − Z = {2, 3, 4}∆ = 1
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 42: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/42.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 8
a(6)
b(3)
c(3)
d(5)
1(-2)
2(-1)
3(0)
4(0)
S
T RS = {a};RT = {3};Z = {a, 1, c, 4, d, 2};T − Z = {3}∆ = 2
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 43: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/43.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 9a(8)
b(5)
c(5)
d(7)
1(-4)
2(-3)
3(0)
4(-2)
S
T
RS = {a};RT = {3};Z = {a, 1, c, 4, 3};
1 2 3 4a 4 6 8 10b 3 2 5 9c 1 7 8 3d 6 4 10 5
34 / 155
![Page 44: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/44.jpg)
A dual method : Example
DefsI let RS ⊆ S and RT ⊆ T be the vertices not covered by M.I Let Z be the set of vertices reachable in −→Gy from RS by a directed
path only following edges that are tight.I If RT ∩ Z 6= ∅, reverse the orientation of a directed path in −→Gy fromRS to RT .
I If RT ∩ Z = ∅, then let:
∆ := min{ci,j − yi − yj : i ∈ Z ∩ S, j ∈ T \ Z}.
I Increase y by ∆ on Z ∩ S and decrease y by ∆ on Z ∩ T .
Example: Iteration 10
a(8)
b(5)
c(5)
d(7)
1(-4)
2(-3)
3(0)
4(-2)
S
T
C = C(c, 1) + C(b, 2) + C(d, 4) + C(a, 3)= 1 + 2 + 5 + 8= 16=
∑i∈{a,b,c,d} yi +
∑j∈{1,2,3,4} yj
= 8 + 5 + 5 + 7− 4− 3 + 0− 2= 16 34 / 155
![Page 45: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/45.jpg)
Pimal/Dual methods
A pair of solutions feasible both in the primal and the dual is optimalif and only if:
xij(cij − ui − vj) = 0
Moreover, given (ui)i∈{1,...,n} and (vj)j∈{1,...,n} solving LSAP for cij isequivalent to solve it for cij = cij − ui − vj . Indeed :∑n
i=1∑nj=1(cij − ui − vj)xij =
∑ni=1∑nj=1 cijxij −
∑ni=1 ui
∑nj=1 xij
−∑nj=1 vj
∑ni=1 xij
=∑ni=1∑nj=1 cijxij −
n∑i=1
ui −n∑j=1
vj︸ ︷︷ ︸constant
The main difference is that if cij contains a sufficient amount of 0, theprimal problem reduces to find a set of n independent 0.
35 / 155
![Page 46: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/46.jpg)
A primal/Dual method
AlgorithmInitialisation: Substract the minimum of each line and then the
minimum of each column.
Step 2: Cover each column containing a selected zero.
Examplevj 0 0 0 0
ui 1 2 3 40 a 4 6 8 100 b 3 2 5 90 c 1 7 8 30 d 6 4 10 5
vj 0 0 0 0ui 1 2 3 44 a 0 2 4 62 b 1 0 3 71 c 0 6 7 24 d 2 0 6 1
vj 0 0 3 1ui 1 2 3 44 a 0 2 1 52 b 1 0 0 61 c 0 6 4 14 d 2 0 3 0
36 / 155
![Page 47: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/47.jpg)
A primal/Dual method
AlgorithmInitialisation: Substract the minimum of each line and then the
minimum of each column.Step 1: Mark each 0 not in the same row or column of a 0
already marked. If n zeros marked: Stop.
Step 2: Cover each column containing a selected zero.
Example0 0 3 11 2 3 4
4 a 0 2 1 52 b 1 0 0 61 c 0 6 4 14 d 2 0 3 0
36 / 155
![Page 48: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/48.jpg)
A primal/Dual methodAlgorithm
Step 2: Cover each column containing a selected zero.I For each non covered zero, mark it by a prime
I If there is a selected zero on the line uncover thecolumn and cover the line
I If there is no selected zero on the line, we do nothave selected enough independent zero. Goto step3.
I If there is no more uncovered zero. Goto step 4.
Example× × ×0 0 3 11 2 3 4
4 a 0 2 1 52 b 1 0 0 61 c 0 6 4 14 d 2 0 3 0
× ×0 0 3 11 2 3 4
4 a 0 2 1 52 b 1 0 0’ 6 ×1 c 0 6 4 14 d 2 0’ 3 0
×0 0 3 11 2 3 4
4 a 0 2 1 52 b 1 0 0’ 6 ×1 c 0 6 4 14 d 2 0’ 3 0 ×
36 / 155
![Page 49: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/49.jpg)
A primal/Dual method
AlgorithmStep 2: Cover each column containing a selected zero.Step 4: Take the minimal value of uncovered elelements found
in step 2. Add this value to each covered line andsubstract it to each non covered column. Return to step1.
Example×0 0 3 11 2 3 4
4 0 2 1 52 1 0 0’ 6 ×1 0 6 4 14 2 0’ 3 0 ×
×0 0 3 11 2 3 4
4 0 2 1 51 2 1 1 7 ×1 0 6 4 13 3 1 4 1 ×
×0 1 4 21 2 3 4
4 a 0 1 0 41 b 2 0 0 6 ×1 c 0 5 3 03 d 3 0 3 0 ×
36 / 155
![Page 50: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/50.jpg)
A primal/Dual method
AlgorithmStep 2: Cover each column containing a selected zero.
I For each non covered zero, mark it by a primeI If there is a selected zero on the line uncover the
column and cover the lineI If there is no selected zero on the line, we do not
have selected enough independent zero. Goto step3.
I If there is no more uncovered zero. Goto step 4.
Example0 1 4 21 2 3 4
4 a 0 1 0’ 4 ×1 b 2 0 0’ 6 ×1 c 0’ 5 3 03 d 3 0’ 3 0 ×
36 / 155
![Page 51: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/51.jpg)
A primal/Dual methodAlgorithm
Step 2: Cover each column containing a selected zero.Step 3: Let z0 be the only uncovered 0 and z1 the selected 0 on
its column. Let zi (i odd ) the 0′ on the line of zi−1 andzi (i even) the selected 0 on the column of zi−1 if itexists (otherwise we stop). The serie zi contains onemore 0′ than selected 0. Exchange 0′ and selected 0.Removes the primes of zeros and the lines and columnscovering. Return to step 1
Example0 1 4 21 2 3 4
4 a 0 1 0’ 4 ×1 b 2 0 0 6 ×
1 c 0’ 5 3 03 d 3 0 3 0 ×
vj 0 1 4 2ui 1 2 3 44 a 0 1 0 41 b 2 0 0 61 c 0 5 3 03 d 3 0 3 0
a
b
c
d
1
2
3
4
S
T
36 / 155
![Page 52: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/52.jpg)
Bipartite Assignement algorithm: MatrixversionInitialisation: Substract the minimum of each line and then the minimum
of each column.Step 1: Mark each 0 not in the same row or column of a 0 already
marked. If n zeros marked: Stop.Step 2: Cover each column by a selected zero.
I For each non covered zero, mark it by a primeI If there is a selected zero on the line uncover the
column and cover the lineI If there is no selected zero on the line, we do not
have selected enough independent zero. Goto step3.
I If there is no more uncovered zero. Goto step 4.Step 3: Let z0 be the only uncovered 0 and z1 the selected 0 on its
column. Let zi (i odd ) the 0′ on the line of zi−1 and zi (ieven) the selected 0 on the column of zi−1 if it exists(otherwise we stop). Exchange 0′ and selected 0. Removesthe primes of zeros and the lines and columns covering.Return to step 1
Step 4: Take the minimal value of uncovered elelements found instep 2. Add this value to each covered line and substract itto each non covered column. Return to step 1.
37 / 155
![Page 53: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/53.jpg)
From Strings to Graphs
I Unfortunately not all problems may be formulated using uni
dimensional relationships between objects.
I We need a more global description of relationships betweenobjects.
38 / 155
![Page 54: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/54.jpg)
Graphs
I G=(V,E)I V set of vertice (or nodes): set of objectsI E set of edges : set of relationships between objects
I Let G = (V,E) be a graph describing a scene coming from asegmentation, a skeleton, a document. . .
I Let G′ = (V ′, E′) be a graph describing a modelI Mean object of a class,I Perfect theoretical object (known or obtained during previous
acquisitions).I Questions:
I Does G,G′ encode a same object ?I Does G describe a part of G′ ?
39 / 155
![Page 55: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/55.jpg)
Graph matching terminology (1/5)
I Let X and Y denote objects/scenes, we want to know if:
X ∼= Y or X j Y.
I Graph isomorphism G = (V,E), G′ = (V ′, E′) (X ∼= Y )I |V | = |V ′| andI it exists φ : V → V ′ bijective such that:
(v1, v2) ∈ E ⇔ (φ(v1), φ(v2)) ∈ E′
I Partial sub graph isomorphism (X j Y )I |V | ≤ |V ′| andI it exists φ : V → V ′ injective such that:
(v1, v2) ∈ E ⇒ (φ(v1), φ(v2)) ∈ E′
40 / 155
![Page 56: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/56.jpg)
Graph matching terminology (2/5)I Sub graph isomorphism
I Same as partial sub graph isomorphism but with the additionalconstraint:
∀(v1, v2) ∈ V 2 (v1, v2) 6∈ E ⇒ (φ(v1), φ(v2)) 6∈ E′
We have thus:(φ(v1), φ(v2)) ∈ E′ ⇔ (v1, v2) ∈ E
I A partial sub graph isomorphism which is not a sub graphisomorphism.
41 / 155
![Page 57: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/57.jpg)
Graph matching terminology (3/5)
I Let X (image) and Y (model), we want to know if it exists Zsuch that:
Z j X and Z j YI Maximum common partial sub graph (mcps).
I Graph of maximal size (in terms of number of vertice)), being apartial sub graph of G and G′.
I Maximum common sub graph (mcs)I same than mcps but isomorphism of sub graph instead of partial
sub graph isomorphism.
42 / 155
![Page 58: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/58.jpg)
Maximal vs maximum sub graph
I (Partial) maximum or maximal sub graph ?I A (partial) common sub graph of G and G′, is said to be
maximal, if we cannot add to it any vertex or edge withoutbreaking the isomorphism property.
I A (partial) common sub graph is said to be maximum if any(partial) common sub graph of G and G′ contains less vertices.
43 / 155
![Page 59: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/59.jpg)
(Sub)graph isomorphisms : typology of ap-proaches
I Graph/Sub graph isomorphismI NP complete problem
I Heuristics to obtain solution (may be sub optimal)I Two approaches:
I Symbolic or algorithmic:I Traverse the set of potential solutions with some heuristics to
reject a priori some solutions.I Good control over the result.I Main actors: Horst Bunke, Marcello Pellilo, Mario Vento,
Pasquale Foggia,. . .I Numerical approaches :
I Define the problem in terms of minimization/maximization of afunction/energy.
I All the tools of numerical optimization are available.I Main actors : Edwin Hancock, Kitler, Sven Dickinson,. . .
44 / 155
![Page 60: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/60.jpg)
From bipartite Assignment to graphmatching
Rem : Image taken from B. Raynal PhD.
45 / 155
![Page 61: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/61.jpg)
Labeled Graphs
I Definitions :I A labeled graph :G = (V,E, µ, ν, Lv, Le){
µ : V → Lv vertex’s label functionν : E → Le edge’s label function
I A labeled sub graph Gs = (Vs, Es, µs, νs, Lv, Le) ofG = (V,E, µ, ν, Lv, Le).I µs and νs restriction of µ and ν to Vs ⊂ V and Ee ⊂ E (with
Es = E ∩ Vs × Vs).I The adjacency matrix M = (mi,j) of a graphG = (V,E, µ, ν, Lv, Le) is defined by:1. ∀i ∈ {1, . . . , n} mi,i = µ(vi)2. ∀(i, j) ∈ {1, . . . , n}2, i 6= j
mi,j ={
ν((vi, vj)) if (vi, vj) ∈ E0 else
46 / 155
![Page 62: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/62.jpg)
Adjacency/Permutation matrices
ia1e1ic 3
e2 ib2or
ia2e1ic 3
e2 ib1
M =123
1 2 3 a e2 e1e2 b 0e1 0 c
, P =123
1 2 3 0 1 01 0 00 0 1
M ′ = PMP t =123
1 2 3 b e2 e1e2 a 0e1 0 c
47 / 155
![Page 63: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/63.jpg)
Graph Isomorphism and Permutation ma-trices
I Two graphs G1 and G2 with matrices M1 and M2 are said to beisomorphic iff it exists a permutation matrix P such that:
M2 = PM1Pt
I It exists a sub graph isomorphism between G1 and G2 iff it existsS ⊂ G2 such that G1 and S are isomorphic(S = (S,E ∩ S × S, µ|S , ν|S , Lv, Le).
I Let M = (mi,j) be a n× n permutation matrix
∀(k,m) ∈ {1, . . . , n}2 Sk,m(M) = (mi,j)i∈{1,...,k},j∈{1,...,m}
I Sk,k(M) is the adjacency matrix of the sub graph restricted tothe k first vertices.
48 / 155
![Page 64: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/64.jpg)
Graph Isomorphism and Permutation ma-trices
I Let G1 and G2 be two graphs with adjacency matrices M1 andM2I M1 : m×m,I M2 : n× n with m ≤ n.
It exists a sub graph isomorphism between G1 and G2 iff it existsa n× n permutation matrix P such that:
M1 = Sm,m(PM2Pt)
I Remark:
M1 = Sm,m(PM2Pt) = Sm,n(P )M2 (Sm,n(P ))t
49 / 155
![Page 65: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/65.jpg)
Matching by State Space Representation(SSR).
I A state: A partial matchingI Method : explore successively the different statesI Distinction between different methods:
I Transition from one state to an otherI Heuristics to avoid infinite loops (consider twice the same state),I Heuristics to restrict the search space
50 / 155
![Page 66: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/66.jpg)
SSR : Algorithme de Cordella 2001 (VF2)
I Notations :I G1 = (V2, E2), G1 = (V2, E2) two oriented graphs,I We seach for either:
I an isomorphism between G1 and G2,I a sub graph isomorphism between G2 and G1 (|V2| ≤ |V1|)
I state s,I M(s) partial matching associated to s,I M1(s) vertices of M(s) in V1,I M2(s) vertices of M(s) in V2I P (s) set of couples ( in V1 × V2) candidates to an inclusion in s,I F (s, n,m) predicate: does the addition of (n,m) to s defines a
partial isomorphism ?I T in1 (s)(T out1 (s)) set of vertices of G1 predecessors (successors) of a
vertex of M1(s).I T in2 (s)(T out2 (s)) set of vertices of G2 predecessors (successors) of a
vertex of M2(s).
51 / 155
![Page 67: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/67.jpg)
VF2 notations: Example
ia?ic� ib����
- id- ie6
i1?i3� i2����
G1 G2
I M(s) = (a, 1)I M1(s) = {a},M2(s) = {1}I T in1 (s) = {b};T out1 (s) = {c};I T in2 (s) = {2};T out2 (s) = {3};
52 / 155
![Page 68: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/68.jpg)
VF2 algorithm
procedure matching(char s)a matching. s0 initial state s.t. M(s0) = ∅
if M(s) contains all vertices of G2 thenreturn M(s)
elsecompute P (s)for each (n,m) ∈ P (s) do
if F (s, n,m) thencompute s′ after the addition of (n,m) to M(s)MATCHING(s’)
end ifend for
end ifRestauration of data structures
end procedure
53 / 155
![Page 69: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/69.jpg)
VF2
I If T out1 (s) and T out2 (s) non empty
P (s) = T out1 (s)× {minT out2 (s)}
min: any order relationship: increasing order of insertion of G2’svertices (avoid to consider 6= paths leading to a same state).
I Else If T in1 (s) et T in2 (s) non empty
P (s) = T in1 (s)× {minT in2 (s)}
I Else if T in1 (s) = T in2 (s) = T out1 (s) = T out2 (s) = ∅
P (s) = (V1 −M1(s))× {min(V2 −M2(s))}
I Rem: If one of the set T in(s) and T out(s) is empty and not theother M(s) cannot lead to a matching.
54 / 155
![Page 70: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/70.jpg)
VF2
F (s, n,m) = Rpred(s, n,m) ∧Rsucc(s, n,m)∧Rin(s, n,m) ∧Rout(s, n,m) ∧Rnew(s, n,m)
I Rpred, Rsucc: Does M(s′) defines a matching ?I Rin, Rout:can we build a matching the step after ?I Rnew: Can I obtain a matching (one day) ?
55 / 155
![Page 71: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/71.jpg)
VF2
I Rpred(s,m, n): predecessors match
(∀n′ ∈M1(s) ∩ Pred(G1, n)∃m′ ∈ Pred(G2,m)|(n′,m′) ∈M(s))∧(∀m′ ∈M2(s) ∩ Pred(G2,m)∃n′ ∈ Pred(G1, n)|(n′,m′) ∈M(s))
n′ ∈M1(s) ↔ m′ ∈M2(s)↓ ↓n ↔ m
56 / 155
![Page 72: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/72.jpg)
VF 2
I Rsucc(s,m, n): successors match
(∀n′ ∈M1(s) ∩ Succ(G1, n)∃m′ ∈ Succ(G2,m)|(n′,m′) ∈M(s))∧(∀m′ ∈M2(s) ∩ Succ(G2,m)∃n′ ∈ Succ(G1, n)|(n′,m′) ∈M(s))
n ↔ m↓ ↓
n′ ∈M1(s) ↔ m′ ∈M2(s)
57 / 155
![Page 73: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/73.jpg)
VF2
I Successors (predecessors) of n and m must match locallyI Rin(s, n,m)
(∣∣T in1 (s) ∩ Succ(G1, n)
∣∣ ≥ ∣∣T in2 (s) ∩ Succ(G2,m)∣∣)∧
(∣∣T in1 (s) ∩ Pred(G1, n)
∣∣ ≥ ∣∣T in2 (s) ∩ Pred(G2,m)∣∣)
n ∈ G1(s′)↓n′ → n′′
∈ T in1 (s) ∈ G1(s)
n ∈ G1(s′)↑n′ → n′′
∈ T in1 (s) ∈ G1(s)
58 / 155
![Page 74: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/74.jpg)
VF2
I Rout(s, n,m)
(|T out1 (s) ∩ Succ(G1, n)| ≥ |T out2 (s) ∩ Succ(G2,m)|)∧(|T out1 (s) ∩ Pred(G1, n)| ≥ |T out2 (s) ∩ Pred(G2,m)|)
n ∈ G1(s′)↓n′ ← n′′
∈ T out1 (s) ∈ G1(s)
n ∈ G1(s′)↑n′ ← n′′
∈ T out1 (s) ∈ G1(s)I Replace ≥ by = for the graph isomorphism
59 / 155
![Page 75: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/75.jpg)
Predicate Rnew
I Successors and predecessors must match outside setsMi(s), T ini (s) et T outi (s), i = 1, 2.I Rnew(s, n,m)
(|N1(s) ∩ Succ(G1, n)| ≥ |N2(s) ∩ Succ(G2,m)|)∧(|N1(s) ∩ Pred(G1, n)| ≥ |N2(s) ∩ Pred(G2,m)|)
I with:I N1(s) = V1 −M1(s)− T in1 (s) ∪ T out1 (s): all that remains to be
seen in G1.I N2(s) = V2 −M2(s)− T in2 (s) ∪ T out2 (s): all that remains to be
seen in G2.
60 / 155
![Page 76: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/76.jpg)
VF2
ia?ic� ib����
- id- ie6
i1?i3� i2����
G1 G2
I M(s) = (a, 1)I M1(s) = {a},M2(s) = {1}I T in1 (s) = {b};T out1 (s) = {c};I T in2 (s) = {2};T out2 (s) = {3};I P (s) = (c, 3)I Pred(G1, c) = {a};Succ(G1, c) = {b, e}I Pred(G2, 3) = {1};Succ(G2, 3) = {2}I F (s, c, 3) = true.
61 / 155
![Page 77: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/77.jpg)
VF2
ia?ic� ib����
- id- ie6
i1?i3� i2����
G1 G2
I M(s′) = {(a, 1), (c, 3)}I M1(s′) = {a, c},M2(s) = {1, 3}I T in1 (s) = {b};T out1 (s) = {b, e};I T in2 (s) = {2};T out2 (s) = {2};I P (s) = (b, 2), (e, 2)I Pred(G1, e) = {c};Succ(G1, e) = {d}I Pred(G2, 2) = {3};Succ(G2, 2) = {1}I (e, 2) violate predicate Rsucc(s′, e, 2). Indeed:
I 1 ∈M2(s) ∩ Succ(G2, 2) or Succ(G1, e) ∩M1(s) = ∅
62 / 155
![Page 78: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/78.jpg)
VF2
ia?ic� ib����
- id- ie6
i1?i3� i2����
G1 G2
I We come up with the matching: M(s′′) = {(a, 1), (c, 3), (b, 2)}and we are done since we cover all the vertices of G2.
63 / 155
![Page 79: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/79.jpg)
VF2
I Complexity:(N = |V1|+ |V2|)VF2 Ullman
Complexity Best case Worse case Best case Worse caseTime O(N2) O(N !N) O(N3) O(N !N2)Space O(N) O(N) O(N3) O(N3)
I Usable for large graphs (up to 1000 vertices).
64 / 155
![Page 80: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/80.jpg)
Association graph
I Let G1 = (V1, E1) and G2 = (V2, E2), the association graphG = (V,E) of G1 and G2 is defined by:I vertices:
V = V1 × V2
I Edges:
E = {((i, h), (j, k)) ∈ V × V |i 6= j, h 6= k and (i, j) ∈ E1 ⇔ (h, k) ∈ E2}
65 / 155
![Page 81: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/81.jpg)
Association graph
Input Graphs1
23
a
c b
d e
4
Association Graph
1-a
1-b
1-c
1-d
1-e
2-a
2-b
2-c
2-d
2-e
1− ∗2− ∗
1-a
1-b
1-c
1-d
1-e
4-a
4-b
4-c
4-d
4-e
1− ∗4− ∗
66 / 155
![Page 82: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/82.jpg)
Association Graph
1
23
4
4,e1,a
1,b
1,c
1,d
1,e
2,a
2,b
2,c
2,d3,a
3,b
3,c
3,d
3,e
4,a
4,b
4,c
4,d
2,e
a
c b
d e
I Sub graph G restricted to {(1, a), (2, b), (3, c)} (blue lines) iscomplete.I All matching are consistent
67 / 155
![Page 83: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/83.jpg)
MCS and cliques
I A complete sub graph of a graph G is called a clique of G.I Maximum/Maximal cliques are defined the same way as common
subgraphsI The clique-number of G, ω(G) is the size (in vertices) of the
maximum clique.I Théorem:
Let G1 and G2 be two graphs and G their association graph. Itexists a bijective relationship between:I maximal/maximum cliques of G andI maximal/maximum common subgraphs of G1 and G2.
I Hence: Computing cliques of the association graph is equivalentto compute the common sub graphs.
68 / 155
![Page 84: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/84.jpg)
SM i, i = 1, 2 algorithms
I Based on a pre computation of all cliques of size i.I i = 1: vertices,I i = 2: couple of vertices.
I Notations:I Neighborhood:
Nj = {k ∈ V | (j, k) ∈ E}I Candidates to the adjunction to clique K:
C0(K) = {j ∈ V −K | ∀k ∈ K (j, k) ∈ E} =⋂k∈K
Nk
69 / 155
![Page 85: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/85.jpg)
SM i: the algorithm
procedure SM i(graph G)Q: clique with cardinal i,K,K∗,maxK = ∅;K∗ = ∅;max=0;while Q 6= ∅ do
H=pop(Q)K=Hwhile C0(K) 6= ∅ do
l =argmaxj∈C0(K) |C0(K) ∩Nj |
K = K ∪ {l}end whileK = K ∪K
if |K|>max thenmax = |K|K∗ = K
end ifend while
end procedure
70 / 155
![Page 86: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/86.jpg)
SM i : Comments
I Try to build as many maximal cliques as initial cliques of size i.I Converge toward a local optimum for each initial clique.I SM2 more efficient but slower than SM1.I SM1_SWAP compromise between SM1 and SM2
I Aim: explore the search space around local optimums.
71 / 155
![Page 87: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/87.jpg)
SM 1_SWAP : Notations
I Candidate to an exchange:
C1(K) = {j ∈ V −K | |Nj ∩K| = |K| − 1}
I j ∈ C1(K) iff only one vertex of K forbids its integration to theclique.
I Let l ∈ C1(K) and kl ∈ K such that (l, kl) 6∈ E. If we add l to K,we must remove kl.
K = K ∪ {l} − {kl}
72 / 155
![Page 88: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/88.jpg)
SM 1_SWAP : Choosing the vertex toadd
l = arg maxj∈C0(K)
|C0(K) ∩Nj | ou l = arg maxj∈C0(K)∪C1(K)
|C0(K) ∩Nj | ?
I The choice of a vertex in C1(K) allows to move away from thelocal optimum but:I This choice does not necessarily improves the optimumI does not allows us to get closer from the convergenceI may induce infinite loops.
I We restrict ourselves to C0(K) :I for a fixed number of iterations (START_SWAP ),I when the number of exchanges is greater than a multiple T of |K|,I when the selected vertex is the one removed by the last exchange
73 / 155
![Page 89: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/89.jpg)
SM 1_SWAP : selection algorithm
function select(G,K, last_swap)l : vertex;if n_swap ≤ T |K| and |K| ≥ START_SWAP then
l = argmaxj∈C0(K)∪C1(K) |C0(K) ∩Nj |if l = last_swap then
l = argmaxj∈C0(K) |C0(K) ∩Nj |end if
elsel = argmaxj∈C0(K) |C0(K) ∩Nj |
end ifreturn l
end functionI We suppose that argmaxj∈C return ∅ if C is empty.
74 / 155
![Page 90: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/90.jpg)
SM 1_SWAP : Algorithm
procedureSM1_SWAP (graphG = (V,E))
int max=0;list of cliques K = ∅,optimal clique K∗ = ∅,queue W = Vwhile W 6= ∅ do
h = pop(W )K = {h}
n_swap = 0;last_swap = ∅
l = select(G,K, last_swap)while l 6= ∅ do
if l ∈ C0(K) thenK = K ∪ {l}
elsen_swap+ = 1last_swap = kl
K = K ∪ {l} − {kl}push(W,kl)
end if
l = select(G,K, last_swap)end whileK = K ∪ {K}if |K|>max then
max = |K|K∗ = K
end ifend while
end procedure
75 / 155
![Page 91: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/91.jpg)
SM 1_SWAP : PerformancesI clique size/ max size: |K|ω(G)
density
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
GRSM1
SM1_SWAP
I Execution times (seconds):
tim
e(s)
density
0
20
40
60
80
100
0 0.2 0.4 0.6 0.8 1
GRSM1
SM1_SWAP
76 / 155
![Page 92: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/92.jpg)
Optimisation methods
1. Encode the graph by a matrix2. Transform a graph problem into the maximisation/minimisation
of some expression (using matrix encoding)3. Choose an optimisation method
77 / 155
![Page 93: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/93.jpg)
Characteristic vector
I Let us consider G = (V,E) et C ⊂ V with |V | = n
I Characteristic vector of C:
xC =
x1...xn
with xi ={ 1|C| if i ∈ C0 otherwise
I Any characteristic vector belongs to the simplex of dim n:
∀C ⊂ V xC ∈ Sn = {x ∈ Rn | etx = 1 et ∀i ∈ {1, . . . , n}xi ≥ 0}
78 / 155
![Page 94: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/94.jpg)
Quadratic Function
I Let AG = (ai,j) the adjacency matrix of a graph G :
ai,j ={
1 if (i, j) ∈ E,0 otherwise.
I and let the function:
g(x) = xtAGx+ 12x
tx = xtAx with{A = AG + 1
2Ix ∈ Sn
I x∗ is a strict maxima of g iff:
∃ε > 0 |{|y − x∗| < ε ⇒ f(y) ≤ f(x∗)|y − x∗| < ε and f(y) = f(x∗) ⇒ y = x∗
79 / 155
![Page 95: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/95.jpg)
Motzkin-Straus(65)-Bomze(97) theorem
I Theorem: Let S ⊂ V and xS its characteristic vector, then:1. S is a maximum clique of G iff xS is a global maximum of g over
Sn. We have then:
ω(G) = 12(1− g(xS))
2. S is a maximal clique of G iff xS is a local maximum of g over Sn.3. Any local maxima (and thus a global one) x of g over Sn is strict
and corresponds to a characteristic vector xS for some S ⊂ V .
80 / 155
![Page 96: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/96.jpg)
Resolution
I Computation of the maxima of:
g(x) = xtAx avec A = AG + 12I
I replication equations:
xi(t+ 1) = xi(t)(Ax(t))ig(x) note
n∑i=1
xi(t) = g(x)g(x) = 1
I A being symmetric:I g(x(t)) is a strictly increasing function of t,I The procesus converge to a stationnary point (xi(t+ 1) = xi(t))
which corresponds to a local maximum of g.
81 / 155
![Page 97: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/97.jpg)
Gibbons, Bomze (97)
I Let us consider AG
= (aij) with ai,j = 1 if (i, j) 6∈ E, 0 otherwise.
AG
= eet −AG − I
I Problem transformation:
f(x) = xtAx = xt(AG
+ 12I)x
= xt(eet −AG − I + 1
2I)x
= xt[eet −
(AG + 1
2I)]x
= xteetx− xt(AG + 1
2I)x
= 1− g(x)
82 / 155
![Page 98: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/98.jpg)
Gibbons, Bomze(97) theorem
I Theorem: Let S ⊂ V and xS its characteristic , then:1. S is a maximum clique of G iff xS is a global minimum of f over
Sn. We have then:ω(G) = 1
2f(x∗)2. S is a maximal clique of G iff xS is a local minimum of f over Sn.3. Any local minimum (and hence global minimum) x of g over Sn is
strict and corresponds to a characteristic vector xS for someS ⊂ V .
83 / 155
![Page 99: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/99.jpg)
Theorem intuition (1/2)
I Problem formulation
f(x) = xtAx = xt(AG
+ 12I)x
=∑ni=1
12x
2i +
∑j|(i,j)6∈E xixj
I for any characteristic vector xC of C ⊂ V :
f(xC) = |C|2|C|2 +
∑ni=1∑j|(i,j)6∈E x
Ci x
Cj
= 12|C| +
∑∑(i,j)∈C2|(i,j)6∈E x
Ci x
Cj
If C is a clique :f(x) = 1
2|C|I The more C is «large» the more, f(x) is «small».
84 / 155
![Page 100: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/100.jpg)
Theorem intuition (2/2)
I Let us suppose that we add to C a single vertex adjacent to allvertices of C but one.
I Let xC′ the resulting characteristic vector:
f(xC′) = 12|C| +
∑∑(i,j)∈C′2|(i,j)6∈E x
C′
i xC′
j
= 12(|C|+1) + 1
(|C|+1)2
I f(xC′) is larger or smaller than f(xC) ?
12(|C|+1) + 1
(|C|+1)2 > 12|C|
m|C|+ 1 + 2 > (|C|+1)2
|C| = |C|+ 2 + 1|C|
m3 > 2 + 1
|C|
85 / 155
![Page 101: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/101.jpg)
Interest of the reformulation
I Application to valuated graphsI maximum Clique:
ω(G) = max{|S| such that S is a clique of G}
I maximum weighted cliqueI Let us consider a vector of weight: w ∈ Rn
ω(G,w) = max{W (S) such thatS is a clique of G}
withW (S) =
∑i∈S
wi
86 / 155
![Page 102: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/102.jpg)
Wighted Cliques
Cliques of maximal (resp. maximum) weights correspond to the local(resp. global) minimums of:
f(x) = xtC(w)x
with
C(w)i,j =
1
2wi If i = j1
2wi + 12wj If i 6= j and (i, j) 6∈ E
0 otherwise
I Ability to include a priori informations !I distance between points,I regions similarities. . .
87 / 155
![Page 103: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/103.jpg)
Resolution
I Problem: minimise over Sn
f(x) = xtAx
I Formulation in terms of «Linear Complementarity Problem»(LCP)I Found y, x such that:
y = qG +MGx ≥ 0, x = [x, xn+1, xn+2], xty = 0
qG =
0...0−1
1
et MG =
(A −e eet 0 0−et 0 0
)
88 / 155
![Page 104: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/104.jpg)
Resolution : Remarks
I LCP: Iterative method based on the choice of a pivot element ateach step
I Locatelli& Pellilo proposed an heuristic method to choose thesepivoting elements → PBH algorithm.
I The PBH algorithm if formelly equivalent to the SM1 algorithm.
89 / 155
![Page 105: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/105.jpg)
Soft Assign algorithm
I We have to determine a permutation matrix between 2 graphs tocheck the existence of an isomorphism or a sub graphisomorphisms.
I The Soft assign algorithm is one of the most famous algorithm tothat aim.
I We consider two graphs G and g with valuated edgesI Ga,b weight of edge (a, b) in GI gi,j weight of edge (i, j) in g
I We consider:I A permutation matrix M (Ma,i = 1) if a is associated to i, 0
otherwise.I A similarity function between edges:
Ca,b,i,j ={
0 Si Ga,b or gi,j is nullc(Ga,b, gi,j) otherwise
with (for example):
c(Ga,b, gi,j) = 1− 3|Ga,b − gi,j |
90 / 155
![Page 106: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/106.jpg)
Soft assign problem
I We would like to minimize :
Ewg(M) = −12
A∑a=1
I∑i=1
A∑b=1
I∑j=1
Ma,iMb,jCa,i,b,j
with A (resp. I ) nb vertices in G (resp. g).I i.e. match a maximum of similar edges
91 / 155
![Page 107: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/107.jpg)
SoftAssign : Illustration
I If Ga,b, gi,j ∈ {1, NULL}, Ca,b,i,j ∈ {0, 1} and
Ewg(M) = −12
A∑a=1
I∑i=1
A∑b=1
I∑j=1
Ma,iMb,jGa,bgi,j
I What we are searching for:
b•
•a
6Ga,b
j•
•i
6gi,j
Mai
Mbj
92 / 155
![Page 108: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/108.jpg)
SoftAssign: simple problem
I Let us consider the small following problem: Given {X1, . . . , Xn}determine {m1, . . . ,mn} such that:I mi = 1 if Xi = maxjXj ,I 0 otherwise.
I equivalent to maximize:n∑i=1
miXi withn∑i=1
mi = 1,mi ∈ {0, 1}
I For any β > 0 let us consider:
mj(β) = eβXj∑ni=1 e
βXi
I We have:
limβ→+∞
mj(β) ={
1 if Xj = maxXi
0 otherwise
93 / 155
![Page 109: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/109.jpg)
SoftAssign : Algorithm 1
function softassign1({X1, . . . , Xn})β ← β0while β < βf do
mi ← eβXj
mi ← mi∑n
i=1mi
Do other parts of the algorithmincrease β
end whilereturn {m1, . . . ,mn}
end functionI We determine the max softly (hence the name soft assign)
94 / 155
![Page 110: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/110.jpg)
SoftAssign : Algorithm 2
I Let now consider a permutation matrix M , between two graphsG and g and one variable Xa,i.
I Maximize according to M :
Eass(M) =A∑a=1
I∑i=1
Ma,iXa,i
I We have no more a single constraint (∑ni=1mi = 1) but two:
I ∀a ∈ {1, . . . , A}∑I
i=1 Ma,i = 1I ∀i ∈ {1, . . . , I}
∑A
a=1 Ma,i = 1
95 / 155
![Page 111: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/111.jpg)
SoftAssign : Algorithm 2
I We normalize iteratively according to lines and columns ≈ weapply algorithm 1 one lines and then on columnsprocedure softassign2(Xai)
Output : Mβ ← β0while β < βf do
Ma,i ← eβXai
repeatMa,i ← Ma,i∑I
j=1Ma,j
Ma,i ← Ma,i∑A
x=1Mx,i
until M convergeDo the remaining part of the algorithmIncrement β
end whileend procedure
I M converges toward a permutation matrix96 / 155
![Page 112: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/112.jpg)
SoftAssign: Towards the (1/2)
I Problem: The matching is not a research of a max
argmaxA∑a=1
I∑i=1
Ma,iXa,i 6= argmin−12
A∑a=1
I∑i=1
A∑b=1
I∑j=1
Ma,iMb,jCa,i,b,j
I Let us considerEwg(M) = − 1
2∑Aa=1
∑Ii=1∑Ab=1∑Ij=1Ma,iMb,jCa,i,b,j as a
function of AI variables.I and apply Taylor to order 1:
Ewg(M) ≈ Ewg(M0) +A∑a=1
I∑i=1
∂Ewg∂Mai
(M)|M=M0(Ma,i −M0a,i)
97 / 155
![Page 113: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/113.jpg)
SoftAssign : Toward the solution
I We have thus:
Ewg(M) = − 12∑Aa=1
∑Ii=1∑Ab=1∑Ij=1Ma,iMb,jCa,i,b,j
≈ Ewg(M0) +∑Aa=1
∑Ii=1
∂Ewg∂Mai
(M0)(Ma,i −M0a,i)
with:∂Ewg∂Ma,i
(M0) = −A∑b=1
I∑j=1
M0b,jCa,i,b,j
98 / 155
![Page 114: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/114.jpg)
Softassign: toward the solution
I Lets define Qa,i = −∂Ewg(M0)∂Ma,i
I we have:
Ewg(M) ≈ Ewg(M0)−∑Aa=1
∑Ii=1Qai(Ma,i −M0
a,i)≈ Cte−
∑Aa=1
∑Ii=1Qa,iMa,i
I Minimizing Ewg(M) becomes equivalent to maximize:
A∑a=1
I∑i=1
Qa,iMa,i
A problem of computation of a max !
99 / 155
![Page 115: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/115.jpg)
Soft Assign: The method
1. Take an initial guess M2. Perform a Taylor decomposition of Ewg(M)3. Perform a softassign corresponding to the computation of the
max of:A∑a=1
I∑i=1
Qa,iMa,i
4. Take the resulting M as result and loop by incrementing βI Remark : We add lines and columns to M in order to transform
inequalities∑Aa=1Ma,i ≤ 1 and
∑Ii=1Ma,i ≤ 1 into equalities →
matrix M .I allows to encode non matched vertices .
100 / 155
![Page 116: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/116.jpg)
SoftAssign: The algorithm
procedure SoftAssign(G, g, βf , β0)Output : β,M
β ← β0Ma,i ← 1 + εwhile β < βf do
repeatQa,i ← −∂Ewg(M)
∂Ma,i
M0a,i ← eβQa,i
repeatM1a,i ←
M0a,i∑I+1
i=1M0a,i
M0a,i ←
M1a,i∑A+1
a=1M1a,i
until M converge or nb iter > I1until M converge or nb iter > I0β ← βrβ
end whilethreshold Ma,i
end procedure 101 / 155
![Page 117: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/117.jpg)
Extension to vertex’s attributes
I Add a function C(1)a,i encoding distances between vertices
Earg(M) = − 12∑Aa=1
∑Ii=1∑Ab=1∑Ij=1Ma,iMb,jC
(2)a,i,b,j
+α∑Aa=1
∑Ii=1Ma,iC
(1)a,i
I is equivalent to add αC(1)a,i to Qa,i in previous algorithm.
102 / 155
![Page 118: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/118.jpg)
Graph Edit distance
I Exactly the same formulation than String edit distance.I d(G1, G2) is the minimal cost of a set of operations transformingG1 into G2 using vertex/edge insertion/deletion/substitutions.
I Solved with A∗ algorithms but with an exponential complexity inthe number of nodes of both graphs.
I Efficient heuristic (and thus sub optimal) algorithms exists basedon the bipartite assignment algorithm.
103 / 155
![Page 119: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/119.jpg)
Edit Paths
Definition (Edit path)Given two graphs G1 and G2 an edit path between G1 and G2 is asequence of node or edge removal, insertion or substitution whichtransforms G1 into G2.
xx x
h -
Removal of twoedges and one node
x xh -
insertion ofan edge
x xh@
@
-
substitution ofa node
x xh@
@
A substitution is denoted u→ v, an insertion ε→ v and a removalu→ ε.
Alternative edit operations such as merge/split have been alsoproposed[Ambauen et al., 2003].
104 / 155
![Page 120: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/120.jpg)
Costs
All paths go to Roma. . . However we are usually only interested bythe shortest one.Let c(.) denote the cost of any elementary operation. The cost of anedit path is defined as the sum of the costs of its elementaryoperations.I All cost are positive: c() ≥ 0,I A node or edge substitution which does not modify a label has a
0 cost: c(l→ l) = 0.
xx x
hG1
-
e1 = | → ε
e2 = − → ε
e3 = • → ε
x xh -
e4 = ε → \
x xh@
@
-
e5 = s-s x xh@
@
G2
If all costs are equal to 1, the cost of this edit path is equal to 5.
105 / 155
![Page 121: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/121.jpg)
Graph edit distance
Definition (Graph edit distance)The graph edit distance between G1 and G2 is defined as the cost ofthe less costly path within Γ(G1, G2). Where Γ(G1, G2) denotes theset of edit paths between G1 and G2.
d(G1, G2) = minγ∈Γ(G1,G2)
∑e∈γ
c(e)
106 / 155
![Page 122: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/122.jpg)
Main concepts
Tree search algorithms explore the space Γ(G1, G2) with someheuristics to avoid to visit unfruitful states. More precisely let usconsider a partial edit path p between G1 and G2. Let:I g(p) the cost of the partial edit path.I h(p) a lower bound of the cost of the remaining part of the path
required to reach G2.
∀γ ∈ Γ(G1, G2), γ = p.q, g(p) + h(p) ≤ dγ(G1, G2)
where dγ(G1, G2) denotes the cost of the edit path γ.
107 / 155
![Page 123: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/123.jpg)
Main concepts
Let UB denote the best approximation of the GED found so far. Ifg(p) + h(p) > UB we have:
∀γ ∈ Γ(G1, G2), γ = p.q, dγ(G1, G2) ≥ g(p) + h(p) > UB
In other terms, all the sons of p will provide a greater approximationof the GED and correspond thus to unfruitful nodes.
108 / 155
![Page 124: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/124.jpg)
Choosing a good lower bound
Let us suppose that n1 and n2 vertices of respectively V1 and V2remain to be assigned. Different choices are possible for functionh[Abu-Aisheh, 2016]:Null function: h(p) = 0,Bipartite Function: h(p) = dlb(G1, G2) (see bellow).
Closer bound but requires a O(max{n1, n2}3)algorithm.
Hausdorff distance h(p) is set to the Hausdroff distance between thesets of n1 and n2 vertices (including their incidentedges). Requires O((n1 + n2)2) computation steps.
109 / 155
![Page 125: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/125.jpg)
A∗ algorithm
1: Input: Two graphs G1 and G2 with V1 = {u1, . . . , un} and V2 = {v1, . . . , vm}2: Output: A minimum edit path between G1 and G23: OPEN={u1 → ε} ∪
⋃w∈V2
{u1 → w}4: repeat5: p = arg minq∈OPEN{g(q) + h(q)}6: if p is a complete edit path then7: return p8: end if9: Complete p by operations on uk+1 or on remaining vertices of V2.10: Add completed paths to OPEN11: until end of times
110 / 155
![Page 126: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/126.jpg)
A∗ algorithm: discussion
I yq q If algorithm A∗ terminates it always returns the optimalvalue of the GED.
I yq q The set OPEN may be as large as the number of edit pathsbetween G1 and G2.
I yq q The algorithm do not return any result before it finds theoptimal solution.
111 / 155
![Page 127: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/127.jpg)
Depth first search algorithm
1: Input: Two graphs G1 and G2 with V1 = {u1, . . . , un} and V2 = {v1, . . . , vm}2: Output: γUB and UB a minimum edit path and its associated cost3:4: (γUB , UBCOST ) = GoodHeuristic(G1, G2)5: initialize OPEN6: while OPEN 6= ∅ do7: p = OPEN.popF irst()8: if p is a leaf (i.e. if all vertices of V1 are mapped) then9: complete p by inserting pending vertices of V210: update (γUB , UBCOST ) if required11: else12: Stack into OPEN all sons q of p such that g(q) + h(q) < UBCOST .13: end if14: end while
112 / 155
![Page 128: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/128.jpg)
Depth first search algorithm
I yq qThe number of pending edit paths in OPEN is bounded by|V1|.|V2|,
I yq qThe initialization by an heuristic allows to discard manybranches,
I This algorithm quickly find a first edit path. It may be tunedinto [Abu-Aisheh, 2016]:I An any time algorithm,I a Parallel or distributed algorithm.
I yq q The computation of the optimal value of the Graph Editdistance may anyway require long processing times.
113 / 155
![Page 129: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/129.jpg)
Independent Edit paths
An element γ ∈ Γ(G1, G2) is potentially infinite by just doing andundoing a given operation (e.g. insert and then delete a node). Allcost being positive such an edit path can not correspond to aminimum:
Definition (Independent Edit path)An independent edit path between two labeled graphs G1 and G2 is anedit path such that:1. No node nor edge is both substituted and removed,2. No node nor edge is simultaneously substituted and inserted,3. Any inserted element is never removed,4. Any node or edge is substituted at most once,
114 / 155
![Page 130: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/130.jpg)
Decomposition of an Edit Path
PropositionThe elementary operations of an independent edit path between twographs G1 and G2 may be ordered into a sequence of removals,followed by a sequence of substitutions and terminated by a sequenceof insertions.
G1-G2
?
6
G1 G2-
R I
S
γ
Note that G1 ∼=sG2
115 / 155
![Page 131: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/131.jpg)
Decomposition of an Edit Path
xx x
hG1
-
e1 = | → ε
e2 = − → ε
e3 = • → ε
x xh -
e4 = ε → \
x xh@
@
-
e5 = s-s x xh@
@
G2
May be reordered into:
xx x
h-
e1 = | → ε
e2 = − → ε
e3 = • → ε
x xh -
e5 = s-s x xh -
e4 = ε → \
x xh@
@
G1 G1 G2 G2R S I
116 / 155
![Page 132: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/132.jpg)
GED Formulation based on G1
Let G1 = (V1, E1) and G2 = (V2, E2). We have:
d(G1, G2) =∑
v∈V1\V1
cvd(v) +∑
e∈E1\E1
ced(e) +∑v∈V1
cvs(v) +∑e∈E1
ces(e)
+∑
v∈V2\V2
cvi(v) +∑
e∈E2\E2
cei(e)
If all costs are constants we have:
d(G1, G2) = (|V1| − |V1|)cvd + (|E1| − |E1|)ced + Vfcvs + Efces+ (|V2| − |V2|)cvi + (|E2| − |E2|)cei
where Vf (resp. Ef ) denotes the number of vertices (resp. edges)substituted with a non zero cost.
117 / 155
![Page 133: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/133.jpg)
GED Formulation based on G1
By grouping constant terms minimize:
d(G1, G2) = (|V1| − |V1|)cvd + (|E1| − |E1|)ced + Vfcvs + Efces+ (|V2| − |V2|)cvi + (|E2| − |E2|)cei
is equivalent to maximize:
M(P ) not.= |V1|(cvd + cvi) + |E1|(ced + cei)− Vfcvs − Efces
We should thus maximize G1 while minimizing Vf and Ef .
If c(u→ v) ≥ c(u→ ε) + c(ε→ v), M(P ) is maximum forVf = Ef = ∅ and G1 is a maximum common sub graph of G1 andG2[Bunke, 1997, Bunke and Kandel, 2000].
118 / 155
![Page 134: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/134.jpg)
Further restriction
Definition (Restricted edit path)A restricted edit path is an independent edit path in which an edgecannot be removed and then inserted.
PropositionIf G1 and G2 are simple graphs, there is a one-to-one mappingbetween the set of restricted edit paths between G1 and G2 and the setof injective functions from a subset of V1 to V2 . We denote by ϕ0 ,the special function from the empty set onto the empty set.
G1-G2
?
6
G1 G2-
R I
ϕ
γ
119 / 155
![Page 135: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/135.jpg)
ε-assignments
I Let V1 and V2 be two sets, with n = |V1| and m = |V2|.I Consider V ε1 = V1 ∪ {ε} and V ε2 = V2 ∪ {ε}.
DefinitionAn ε-assignment from V1 to V2 is a mapping ϕ : V ε1 → P(V ε2 ),satisfying the following constraints:
∀i ∈ V1 |ϕ(i)| = 1∀j ∈ V2 |ϕ−1(j)| = 1ε ∈ ϕ(ε)
An ε assignment encodes thus:1. Substitutions: ϕ(i) = j with (i, j) ∈ V1 × V2.2. Removals: ϕ(i) = ε with i ∈ V1 .3. Insertions: j ∈ ϕ−1(ε) with j ∈ V2.
120 / 155
![Page 136: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/136.jpg)
From assignments to matrices
I An ε−assignment can be encoded in matrix form:X = (xi,k)(i,k)∈{1,...,n+1}×{1,...,m+1} with
∀(i, k) ∈ {1, . . . , n+ 1} × {1, . . . ,m+ 1} xi,k ={
1 if k ∈ ϕ(i)0 else
where n+ 1 denotes the ε of V1 and m+ 1 the one of V2.I We have:
∀i= 1, . . . , n,m+1∑k=1
xi,k = 1 (|ϕ(i)| = 1)
∀k= 1, . . . ,m,n+1∑j=1
xj,k = 1 (|ϕ−1(k)| = 1)
xn+1,m+1 = 1 (ε ∈ ϕ(ε))∀(i, j) xi,j ∈ {0, 1}
121 / 155
![Page 137: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/137.jpg)
From functions to matrices
u1
u2
u3
ε
v1
v2
ε
V1
V2
x =
v1 v2 | ε
u1 1 0 | 0u2 0 1 | 0u3 0 0 | 1
ε 0 0 | 1
I The set of permutation matrices encoding ε−assignments is
called the set of ε-assignment matrices denoted by An,m.I Usual assignments are encoded by larger (n+m)× (n+m)
matrices[Riesen, 2015].
122 / 155
![Page 138: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/138.jpg)
Back to edit paths
I Let us consider two simple graphs G1 = (V1, E1) andG2 = (V2, E2) with |V1| = n and |V2| = m.
PropositionThere is a one-to-one relation between the set of restricted edit pathsfrom G1 to G2 and An,m.
Note that |An,m| =∑min(n,m)p=0 p!
(np
)(mp
). Hence we have (with
n ≤ m):(n+m)!
n! ≤ |An,m| ≤ (n+m)!
These bounds also bound the number of restricted edit paths betweenG1 and G2.
TheoremAny non-infinite value of 1
2xT∆x + cTx corresponds to the cost of arestricted edit path. Conversely the cost of any restricted edit pathmay be written as 1
2xT∆x + cTx with the appropriate x.123 / 155
![Page 139: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/139.jpg)
Costs of Node assignments
C =
c(u1 → v1) · · · c(u1 → vm) c(u1 → ε). . .
...... c(ui → vj)
... c(ui → ε). . .
...c(un → v1) . . . c(un → vm) c(un → ε)c(ε→ v1) c(ε→ vi) c(ε→ vm) 0
I c = vect(C)I Alternative factorizations of cost matrices
exist[Serratosa, 2014, Serratosa, 2015].
124 / 155
![Page 140: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/140.jpg)
Cost of edges assignments
I Let us consider a (n+ 1)(m+ 1)× (n+ 1)(m+ 1) matrix D suchthat:
dik,jl = ce(i→ k, j→ l)
with:
(i, j) (k, l) edit operation cost ce(i→ k, j→ l)∈ E1 ∈ E2 substitution of (i, j) by (k, l) ce((i, j)→ (k, l))∈ E1 6∈ E2 removal of (i, j) ce((i, j)→ ε)6∈ E1 ∈ E2 insertion of (k, l) into E1 ce(ε→ (k, l))6∈ E1 6∈ E2 do nothing 0
I ∆ = D if both G1 and G2 are undirected andI ∆ = D + DT else.I Matrix ∆ is symmetric in both cases.
125 / 155
![Page 141: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/141.jpg)
Graph edit distance and assignments
I We have thus:
GED(G1, G2) = min{
12xT∆x + cTx | x∈ vec[An,m]
}I Matrix ∆ is non convex with several local minima. The problem
is thus NP-complete.
126 / 155
![Page 142: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/142.jpg)
Let us approximate
I One solution to solve this quadratic problem consists in droppingthe quadratic term. We hence get:
d(G1, G2) ≈ min{
cTx | x∈ vec[An,m]}
= minn+m∑i=1
n+m∑j=1
ci,jxi,j
I This problem is an instance of a bipartite graph matchingproblem also called linear sum assignment problem.
127 / 155
![Page 143: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/143.jpg)
Bipartite Graph matching
I Approximations of the Graph edit distance defined usingbipartite Graph matching methods are called bipartite Graphedit distances (BP-GED).
I Different methods, such as Munkres orJonker-Volgerand[Burkard et al., 2012] allow to solve thisproblem in cubic time complexity.
I The general procedure is as follows:1. compute:
x = arg min cTx2. Deduce the edge operations from the node operations encoded by
x.3. Return the cost of the edit path encoded by x.
128 / 155
![Page 144: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/144.jpg)
Matching Neighborhoods
I The matrix C defined previoulsy only encodes node information.I Idea: Inject edge information into matrix C. Let
di,j = minx
n+m∑k=1
c(ik → jl)xk,l
the cost of the optimal mapping of the edges incident to i ontothe edge incident to j.
I Let :c∗i,j = c(ui → vj) + di,j and x = arg min(c∗)Tx
I The edit path deduced from x defines an edit cost dub(G1, G2)between G1 and G2.
129 / 155
![Page 145: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/145.jpg)
Matching Neighborhoods
I We may alternatively consider:
dlb(G1, G2) =n+m∑i=1
n+m∑j=1
(c(ui → vj) + 1
2di,j)xi,j
I The resulting distances satisfy[Riesen, 2015]:
dlb(G1, G2) ≤ d(G1, G2) ≤ dub(G1, G2)
130 / 155
![Page 146: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/146.jpg)
Matching larger structure
I The upper bound provided by dub(G1, G2) may be large,especially for large graphs. So a basic idea consists in enlargingthe considered substructures:1. Consider both the incident edges and the adjacent nodes.2. Associate a bag of walks to each vertex and define C as the cost
of matching these bags [Gaüzère et al., 2014].3. Associate to each vertex a subgraph centered around it and define
C as the optimal edit distance between thesesubgraphs [Carletti et al., 2015].
4. Consider rings of nodes centered around eachvertex [Blumenthal et al., 2018],
5. . . .I All these heuristics provide an upper bound for the Graph edit
distance.I But: up to a certain point the linear approximation of a
quadratic problem reach its limits.
131 / 155
![Page 147: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/147.jpg)
Improving the initial edit path
I One idea to improve the results of the linear approximation ofthe GED consists in applying a post processing stage.
I Two ideas have been proposed:1. By modifying the initial cost matrix and recomputing a new
assignment.2. By swapping elements in the assignment returned by the linear
algorithm.
132 / 155
![Page 148: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/148.jpg)
BP-Iterative
I q consecutive trials to improve the initial guess provided by abipartite algorithm.
1: Build a Cost matrix C∗2: Compute x = arg min(c∗)T x3: Compute dbest = dx(G1, G2)4: for i=1 to q do5: determine node operation ui → vj with highest cost6: set c∗i,j = +∞ . prevent assignment ui → vj7: compute a new edit path γ on modified matrix C∗.8: if dγ(G1, G2) < dbest then9: dbest = dγ(G1, G2)10: end if11: end for
133 / 155
![Page 149: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/149.jpg)
Alternative strategies
I BP-Iterative never reconsiders an assignment.I Riesen[Riesen, 2015] proposed an alternative procedure calledBP − Float which after each new forbidden assignment:1. reconsider the entries previously set to +∞,2. Restore them if the associated edit cost decreases
I An alternative search procedure is based on Geneticalgorithms[Riesen, 2015]:1. Build a population by randomly forbidding some entries of C∗,2. Select a given percentation of the population whose cost matrix is
associated to the lowest edit distance,3. Cross the population by forbidding an entry if this entry is
forbidden by one of the two parent.4. Go back to step 2.
134 / 155
![Page 150: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/150.jpg)
Swapping assignments
I Forbidding entries of matrix C∗ requires to recompute a newassignment from scrach.
I The basic idea of the swapping strategy consits to replaceassignments:
ui → vkuj → vl
)by(ui → vluj → vk
I Or in matrix terms, replacing
xi,k = xj,l = 1;xi,l = xj,k = 0
)by(xi,l = xj,k = 1;xi,k = xj,l = 0.
135 / 155
![Page 151: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/151.jpg)
BP-Greedy-Swap
1: swapped=true2: while swapped do3: Search for indices (i, j, k, l) such that xi,k = xj,l = 14: costorig = c∗i,k + c∗j,l5: costswap = c∗i,l + c∗j,k6: if |costorig − costswap| < θcostorig then7: x′ = swap(x)8: if dx′ (G1, G2) < dbest then9: dbest = dx′ (G1, G2)10: best swap=x’; swapped=true11: end if12: end if13: if we swapped for at least one (i, j) ∈ V 2
1 then update x according to bestswap.
14: end while
136 / 155
![Page 152: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/152.jpg)
Discussion
I BP −Greedy − Swap as BP − Iterative never reconsiders aswap,
I An alternative exploration of the space of solutions may beperformed using Genetic algorithms[Riesen, 2015].
I Or by a tree search:I This procedure is called Beam searchI Each node of the tree is defined by (x, q, dx(G1, G2)) where x is
an assignment and q the depth of the node.I Nodes are sorted according to:
1. their depth,2. their cost.
137 / 155
![Page 153: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/153.jpg)
BP-Beam
1: Build a Cost matrix C∗2: Compute x = arg min(c∗)T x and dx(G1, G2)3: OPEN = {(x, 0, dx(G1, G2))}4: while OPEN 6= ∅ do5: remove first tree node in OPEN: (x, q, dx(G1, G2))6: for j=q+1 to n+m do7: x’=swap(x,q+1,j)8: OPEN = OPEN ∪ {(x′, q + 1, dx′ (G1, G2))}9: if dx′ (G1, G2) < dbest then10: dbest = dx′ (G1, G2)11: end if12: end for13: while size of OPEN > b do14: Remove tree nodes with highest value dx from OPEN15: end while16: end while
138 / 155
![Page 154: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/154.jpg)
From linear to quadratic optimization
I Improvements of Bipartite matching explore the space of editpaths around an initial solution.
I In order to go beyond these results, this search must be guidedby considering the real quadratic problem.
d(G1, G2) = 12xT∆x + cTx
139 / 155
![Page 155: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/155.jpg)
Justice and Hero
This work[Justice and Hero, 2006]:I Proposes a quadratic formulation of the Graph Edit distance,I Designed for Graphs with unlabeled edges by augmenting graphs
with ε nodes,I Solved by relaxing the integer constraint xi,j ∈ {0, 1} toxi,j ∈ [0, 1] and then solving it using interior point method.
I Propose a linear approximation corresponding to bipartitematching.
140 / 155
![Page 156: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/156.jpg)
Neuhauss and Bunke
In this work[Neuhaus and Bunke, 2007]I Also a quadratic formulation but with node operations restricted
to substitution with the induced operations on edges.I Works well mainly for graphs having close sizes with a cost of
substitution lower than a cost of removal plus a cost of insertion.
141 / 155
![Page 157: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/157.jpg)
From quadratic to linear problems
I Let us consider a quadratic problem:
xTQx =n∑i=1
n∑j=1
qi,jxixj
I Let us introduce yn∗i+j = xixj we get:
xTQx =n2∑k=1
qkyk
with an appropriate renumbering of Q’s elements. Hence a LinearSum Assignment Problem with additional constraints.
I Note that the Hungarian algorithm can not be applied due toadditional constraints. We come back to tree based algorithms.
I This approach has been applied to the computation of the exactGraph Edit Distance by[Lerouge et al., 2016]
142 / 155
![Page 158: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/158.jpg)
Franck Wolfe
I Start with an good Guess x0:x = x0while a fixed point is not reached do
b∗ = arg min{xT∆ + cT )b, b ∈ {0, 1}(n+m)2}t∗ = line search between x and b∗x = x+ t∗(b∗ − x)
end whileI b∗ minimizes a sum of:
(xT∆ + cT )i,j = ci,j +n+m∑k,l
di,j,k,lxk,l
which may be understood as the cost of mapping i to j given theprevious assignment x.
I The distance decreases at each iteration. Moreover,I at each iteration we come back to an integer solution,
143 / 155
![Page 159: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/159.jpg)
Graduated NonConvexity and ConcativityProcedure (GNCCP)
I Consider [Liu and Qiao, 2014]:
Sη(x) = (1− |ζ|)S(x) + ζxTx with S(x) = 12x
T∆x+ cTx
where ζ ∈ [−1, 1].
{ζ = 1 : Convex objective functionζ = −1 : Concave objective function
I The algorithm tracks the optimal solution from a convex to aconcave relaxation of the problem.
144 / 155
![Page 160: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/160.jpg)
From ζ = 1 to ζ = 0
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
145 / 155
![Page 161: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/161.jpg)
GNCCP Algorithm
ζ = 1, d = 0.1, x = 0while (ζ > −1) and (x 6∈ An,m) do
Q = 12 (1− |ζ|)∆ + ζI
L = (1− |ζ|)cx = FW (x,Q,L)ζ = ζ − d
end while
146 / 155
![Page 162: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/162.jpg)
Experiments
Dataset Number of graphs Avg Size Avg Degree PropertiesAlkane 150 8.9 1.8 acyclic, unlabeledAcyclic 183 8.2 1.8 acyclicMAO 68 18.4 2.1PAH 94 20.7 2.4 unlabeled, cyclesMUTAG 8 × 10 40 2
147 / 155
![Page 163: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/163.jpg)
Experiments
Algorithm Alkane Acyclicd t d t
A∗ 15.3 1.29 16.7 6.02[Riesen and Bunke, 2009] 37.8 10−4 33.3 10−4
[Gaüzère et al., 2014] 36.0 0.02 31.8 0.02FWRandom init 19.9 0.02 22.2 0.01mFW40 Random Init 15.3 0.1 16.74 0.07
[Neuhaus and Bunke, 2007] 20.5 0.07 25.7 0.042GNCCP 16.6 0.12 18.7 0.32
148 / 155
![Page 164: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/164.jpg)
Experiments
Algorithm MAO PAHd t d t
[Riesen and Bunke, 2009] 95.7 10−3 135.2 10−3
[Gaüzère et al., 2014] 85.1 1.48 125.8 2.6FWInit [Riesen and Bunke, 2009] 38.4 0.04 50.0 0.09FWInit [Gaüzère et al., 2014] 38.7 1.53 46.8 2.68
mFWInit [Riesen and Bunke, 2009] 31.4 0.4 31.5 0.7mFWInit [Gaüzère et al., 2014] 32.4 1.75 29.8 2.96
[Neuhaus and Bunke, 2007] 59.1 7 52.9 8.20GNCCP 34.3 9.23 34.2 14.44
149 / 155
![Page 165: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/165.jpg)
Graph Edit distance Contest
I Two characteristics of a Graph edit distance algorithm:I Mean Deviation:
deviation_scorem = 1#subsets
∑S∈subsets
devmSmax_devS
I Mean execution time:
time_scorem = 1#subsets
∑S∈subsets
timemSmax_timeS
150 / 155
![Page 166: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/166.jpg)
Graph Edit distance Contest
I Several Algorithms (all limited to 30 seconds):I Algorithms based on a linear transformation of the quadratic
problem solved by integer programming:I F2 (•),I F24threads(�),I F2LP ((♦, relaxed problem)
I Algorithms based on Depth first search methods:I DF(5),I PDFS(/),I DFUP(4).
I Beam Search: BS (�)I IPFP[Gaüzère et al., 2014] : QAPE (+)I Bipartite matching[Gaüzère et al., 2014]: LSAPE(×)
151 / 155
![Page 167: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/167.jpg)
Average speed-deviation scores on MUTAsubsets
I Two settings of editing costsvertices edges
cs cd ci cs cd ci
Setting 1 2 4 4 1 2 2Setting 2 6 2 2 3 1 1
152 / 155
![Page 168: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/168.jpg)
Average speed-deviation scores on GR-EYC’s subsets
I Setting:
vertices edges
cs cd ci cs cd ci
Setting 2 4 4 1 1 1
153 / 155
![Page 169: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/169.jpg)
Conclusion
I Bipartite Graph edit distance has re initiated a strong interest onGraph edit Distance from the research community1. It is fast enough to process large graph databases,2. It provides a reasonable approximation of the GED.
I More recently new quadratic solvers for the GED have emerged.1. They remain fast (while slower than BP-GED),2. They strongly improve the approximation or provide exact
solutions.
154 / 155
![Page 170: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/170.jpg)
Bibliography
155 / 155
![Page 171: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/171.jpg)
Abu-Aisheh, Z. (2016). Anytime and distributed Approaches for Graph Matching.PhD thesis, Université François Rabelais, Tours.
Ambauen, R., Fischer, S., and Bunke, H. (2003). Graph edit distance with nodesplitting and merging, and its application to diatom identification. In GraphBased Representations in Pattern Recognition: 4th IAPR InternationalWorkshop, GbRPR 2003 York, UK, June 30 – July 2, 2003 Proceedings, pages95–106, Berlin, Heidelberg. Springer Berlin Heidelberg.
Blumenthal, D., Bougleux, S., Gamper, J., and Brun, L. (2018). Ring basedapproximation of graph edit distance. In Proceedins of SSPR’2018, Beijing.IAPR, Springer.
Bunke, H. (1997). On a relation between graph edit distance and maximumcommon subgraph. Pattern Recogn. Lett., 18(9):689–694.
Bunke, H. and Kandel, A. (2000). Mean and maximum common subgraph of twographs. Pattern Recogn. Lett., 21(2):163–168.
Burkard, R., Dell’Amico, M., and Martello, S. (2012). Assignment Problems:Revised Reprint. Society for Industrial and Applied Mathematics.
Carletti, V., Gaüzère, B., Brun, L., and Vento, M. (2015). Graph-BasedRepresentations in Pattern Recognition: 10th IAPR-TC-15 InternationalWorkshop, GbRPR 2015, Beijing, China, May 13-15, 2015. Proceedings, chapterApproximate Graph Edit Distance Computation Combining Bipartite Matchingand Exact Neighborhood Substructure Distance, pages 188–197. SpringerInternational Publishing, Cham.
155 / 155
![Page 172: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/172.jpg)
Gaüzère, B., Bougleux, S., Riesen, K., and Brun, L. (2014). Structural,Syntactic, and Statistical Pattern Recognition: Joint IAPR InternationalWorkshop, S+SSPR 2014, Joensuu, Finland, August 20-22, 2014. Proceedings,chapter Approximate Graph Edit Distance Guided by Bipartite Matching ofBags of Walks, pages 73–82. Springer Berlin Heidelberg, Berlin, Heidelberg.
Justice, D. and Hero, A. (2006). A binary linear programming formulation of thegraph edit distance. IEEE Transactions on Pattern Analysis and MachineIntelligence, 28(8):1200–1214.
Lerouge, J., Abu-Aisheh, Z., Raveaux, R., Héroux, P., and Adam, S. (2016).Exact graph edit distance computation using a binary linear program. InRobles-Kelly, A., Loog, M., Biggio, B., Escolano, F., and Wilson, R., editors,Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPRInternational Workshop, S+SSPR 2016, Mérida, Mexico, November 29 -December 2, 2016, Proceedings, pages 485–495, Cham. Springer InternationalPublishing.
Liu, Z. and Qiao, H. (2014). GNCCP - graduated nonconvexityand concavityprocedure. IEEE Trans. Pattern Anal. Mach. Intell., 36(6):1258–1267.
Neuhaus, M. and Bunke, H. (2007). A quadratic programming approach to thegraph edit distance problem. In Escolano, F. and Vento, M., editors,Graph-Based Representations in Pattern Recognition: 6th IAPR-TC-15International Workshop, GbRPR 2007, Alicante, Spain, June 11-13, 2007.Proceedings, pages 92–102, Berlin, Heidelberg. Springer Berlin Heidelberg.
155 / 155
![Page 173: Structural Pattern Recognition - GREYC](https://reader030.vdocument.in/reader030/viewer/2022012413/616d2bedabcd364b89664755/html5/thumbnails/173.jpg)
Riesen, K. (2015). Structural Pattern Recognition with Graph Edit Distance.Springer.
Riesen, K. and Bunke, H. (2009). Approximate graph edit distance computationby means of bipartite graph matching. Image and Vision Computing, 27(7):950 –959. 7th IAPR-TC15 Workshop on Graph-based Representations (GbR 2007).
Serratosa, F. (2014). Fast computation of bipartite graph matching. PatternRecognition Letters, 45:244–250.
Serratosa, F. (2015). Speeding up fast bipartite graph matching through a newcost matrix. Int. Journal of Pattern Recognition, 29(2).
155 / 155