inexact graph matching using a hierarchy of …...inexact graph matching using a hierarchy of...

17
Computational Visual Media DOI 10.1007/s41095-015-0030-4 Vol. 1, No. 4, December 2015, 291–307 Research Article Inexact graph matching using a hierarchy of matching processes Paul Morrison 1 , Ju Jia Zou 1 ( ) c The Author(s) 2015. This article is published with open access at Springerlink.com Abstract Inexact graph matching algorithms have proved to be useful in many applications, such as character recognition, shape analysis, and image analysis. Inexact graph matching is, however, inherently an NP-hard problem with exponential computational complexity. Much of the previous research has focused on solving this problem using heuristics or estimations. Unfortunately, many of these techniques do not guarantee that an optimal solution will be found. It is the aim of the proposed algorithm to reduce the complexity of the inexact graph matching process, while still producing an optimal solution for a known application. This is achieved by greatly simplifying each individual matching process, and compensating for lost robustness by producing a hierarchy of matching processes. The creation of each matching process in the hierarchy is driven by an application-specific criterion that operates at the subgraph scale. To our knowledge, this problem has never before been approached in this manner. Results show that the proposed algorithm is faster than two existing methods based on graph edit operations. The proposed algorithm produces accurate results in terms of matching graphs, and shows promise for the application of shape matching. The proposed algorithm can easily be extended to produce a sub-optimal solution if required. Keywords graph matching; inexact graph matching; graph edit distance; graph edit operations; shape matching 1 School of Computing, Engineering and Mathematics, Western Sydney University, Locked Bag 1797, Penrith, NSW 2751, Australia. E-mail: P. Morrison, [email protected]; J. J. Zou, J.Zou@ westernsydney.edu.au ( ). Manuscript received: 2015-10-26; accepted: 2015-11-24 1 Introduction Graphs have proved to be extremely useful and versatile tools in the areas of image processing, computer vision, and pattern recognition. The versatility of graphs stems from their ability to represent virtually any kind of information in a way that is amenable to various forms of interpretation and manipulation. Graph matching is the process of forming correspondences between parts of one graph and parts of another graph. Graph matching has been used in many applications, particularly in the field of visual media processing. They include 2D and 3D image analysis (e.g., shape recognition), interactive image editing (e.g., patch-based image synthesis), document processing (e.g., handwritten character recognition), biometric identification (e.g., human face recognition), image databases (e.g., image retrieval), video analysis (e.g., object tracking), and biomedical applications (e.g., identification of coronary arteries from medical images) [1–4]. A typical problem in shape recognition can be tackled in three steps, namely, the decomposition of a shape into parts, the representation of the parts and their relations using a graph, and the search for correspondence between this graph and that of a known object. In this case, the skeleton of the shape is often used as the graph for shape representation and matching [5]. For a typical problem in patch- based image synthesis, an image is divided into regions and then described by a graph where nodes represent the regions and edges represent spatial relations between the regions. The image can then be modified by replacing some of its regions by suitable regions obtained from an image database using contextual graph matching [2]. The image can 291

Upload: others

Post on 30-Jan-2020

18 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

Computational Visual MediaDOI 10.1007/s41095-015-0030-4 Vol. 1, No. 4, December 2015, 291–307

Research Article

Inexact graph matching using a hierarchy of matchingprocesses

Paul Morrison1, Ju Jia Zou1 (�)

c© The Author(s) 2015. This article is published with open access at Springerlink.com

Abstract Inexact graph matching algorithms haveproved to be useful in many applications, such ascharacter recognition, shape analysis, and imageanalysis. Inexact graph matching is, however,inherently an NP-hard problem with exponentialcomputational complexity. Much of the previousresearch has focused on solving this problem usingheuristics or estimations. Unfortunately, many of thesetechniques do not guarantee that an optimal solutionwill be found. It is the aim of the proposed algorithmto reduce the complexity of the inexact graph matchingprocess, while still producing an optimal solution fora known application. This is achieved by greatlysimplifying each individual matching process, andcompensating for lost robustness by producing ahierarchy of matching processes. The creation ofeach matching process in the hierarchy is driven byan application-specific criterion that operates at thesubgraph scale. To our knowledge, this problem hasnever before been approached in this manner. Resultsshow that the proposed algorithm is faster than twoexisting methods based on graph edit operations.The proposed algorithm produces accurate resultsin terms of matching graphs, and shows promise forthe application of shape matching. The proposedalgorithm can easily be extended to produce asub-optimal solution if required.

Keywords graph matching; inexact graph matching;graph edit distance; graph edit operations;shape matching

1 School of Computing, Engineering and Mathematics,Western Sydney University, Locked Bag 1797,Penrith, NSW 2751, Australia. E-mail: P. Morrison,[email protected]; J. J. Zou, [email protected] (�).

Manuscript received: 2015-10-26; accepted: 2015-11-24

1 Introduction

Graphs have proved to be extremely useful andversatile tools in the areas of image processing,computer vision, and pattern recognition. Theversatility of graphs stems from their ability torepresent virtually any kind of information in a waythat is amenable to various forms of interpretationand manipulation.

Graph matching is the process of formingcorrespondences between parts of one graph andparts of another graph. Graph matching has beenused in many applications, particularly in the fieldof visual media processing. They include 2D and 3Dimage analysis (e.g., shape recognition), interactiveimage editing (e.g., patch-based image synthesis),document processing (e.g., handwritten characterrecognition), biometric identification (e.g., humanface recognition), image databases (e.g., imageretrieval), video analysis (e.g., object tracking),and biomedical applications (e.g., identification ofcoronary arteries from medical images) [1–4]. Atypical problem in shape recognition can be tackledin three steps, namely, the decomposition of ashape into parts, the representation of the partsand their relations using a graph, and the searchfor correspondence between this graph and that of aknown object. In this case, the skeleton of the shapeis often used as the graph for shape representationand matching [5]. For a typical problem in patch-based image synthesis, an image is divided intoregions and then described by a graph where nodesrepresent the regions and edges represent spatialrelations between the regions. The image can thenbe modified by replacing some of its regions bysuitable regions obtained from an image databaseusing contextual graph matching [2]. The image can

291

Page 2: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

292 Paul Morrison, Ju Jia Zou

also be augmented by adding extra regions based ongraph matching [4]. Interested readers are referredto relevant review papers, e.g., Refs. [1, 3], for otherproblems in visual media processing which can beposed as graph matching problems.

A graph matching algorithm can generally beplaced into one of three broad categories: graphisomorphism, subgraph isomorphism, and inexactgraph matching. If the two graphs being matchedare not identical, correspondences cannot be formedby a graph isomorphism algorithm. Subgraphisomorphism relaxes this constraint somewhat, andallows matching of one graph with a subgraphof another graph. However, the subgraph mustrepresent the other graph exactly for a matchto be made. Occlusion, articulation, and otherforms of noise are often present in real-worldapplications, which only the third form of graphmatching, inexact graph matching, is robust enoughto resolve. Rather than indicating whether a matchexists or not, inexact graph matching outputs somekind of similarity measure in addition to formingcorrespondences between graph nodes and/or edges.Inexact graph matching is the most useful andcommonly-used form of graph matching. Inexactgraph matching is the topic of this paper.

Tree search algorithms are commonly used toperform inexact graph matching. In these methods,a tree (state space) is constructed, where eachstate represents a partial mapping between nodesand/or edges in the two graphs being matched. Thegoal is to find the state that has the lowest cost,which represents the optimal match between the twographs.

The first method using tree search techniqueswas developed by Tsai and Fu [6, 7]. Here, theconcepts of graph edit cost and graph deformationare introduced. To guarantee that an optimal matchis always found, a large number of states will oftenneed to be searched. To reduce the number of statessearched, Tsai and Fu utilise any available heuristicinformation to guide the search. For every child stateof a given state, they estimate a lower bound onthe cost from that child state to a goal state. Thisinformation is then used to prioritise the searchingof different branches in the tree.

Various modifications to this idea weresubsequently proposed, in both branch-and-

bound algorithms and look-ahead algorithms suchas A* [6, 8]. Such algorithms have previouslybeen used in artificial intelligence [9]. While suchestimates using heuristics can drastically reducethe time taken to search the state space, theyoften do not guarantee that the optimal solutionwill be found. Other techniques have also beenapplied to the inexact graph matching problem,such as relaxation labeling [10] and graduatedassignment [11]. While many of these algorithmsdo not guarantee that an optimal solution will befound, they are often very fast (they typically run inpolynomial-time). We refer readers to Refs. [12–14]for more comprehensive reviews of inexact graphmatching algorithms.

Optimal inexact graph matching is an NP-hard problem, with expected exponential-timecomputational complexity. As in many graph-relatedtasks of high computational complexity, inexactgraph matching algorithms often trade optimalityfor faster time, and vice versa. Finding an optimalsolution may not be critical in many applications.In other applications, however, an optimal solutionwill be desirable [15]. Where optimality is required,many inexact graph matching algorithms will onlybe of practical use when matching graphs with afew nodes and/or edges. The high computationalcomplexity of current optimal methods poses asignificant problem, which we attempt to addressin this paper. Indeed, there has been some recentinterest in reducing the computational complexityof inherently NP-hard problems, particularly in thefield of graph theory [16–20].

We presume that the optimal solution not onlydepends on the properties of the graphs beingmatched, but also depends on the applicationof the graph matching algorithm. This is oftenimplemented by embedding application-specificattributes in the nodes and edges of the graphs, butthis is on a local scale only. A hierarchical matchingscheme is proposed, that includes an application-specific criterion that operates on the subgraphscale. This scheme allows an optimal match tobe found for the given application. The proposedalgorithm is based on graph edit operations, butis less computationally intensive than the typicalmethod based on graph edit operations. Theproposed algorithm does not require heuristics or

292

Page 3: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

Inexact graph matching using a hierarchy of matching processes 293

estimations, but rather allows optimizations throughthe use of more application-specific information. Toour knowledge, the problem of high computationalcomplexity in inexact graph matching has neverbefore been approached in this way.

This paper is organised in the following manner.In Section 2, the proposed algorithm is describedby building on the existing graph matching theory.Section 3 presents a comparison between thecomputational complexities of the proposed methodand an existing method that is typical of manyprevious inexact graph matching algorithms. Resultsare given in Section 4, and finally, conclusions andopportunities for further research are discussed inSection 5.

2 Hierarchical graph matching

2.1 Graphs

A graph is defined by a tuple g = (V,E, α, β) where• V is the finite set of vertices (nodes),• E is the finite set of edges,• α : V → L is the node labeling function, and• β : E → L is the edge labeling function.

L is a finite alphabet of labels for nodes and edges.Labels, or attributes, are values derived from theapplication-specific information on which the graphis based. Assigning labels to nodes and edges, usingα and β, respectively, adds semantic informationto the graph representation. Edges are directed;the edge e = (x, y) originates at node x ∈ V andterminates at node y ∈ V . In this paper, thenotation exy is used to denote an edge originatingat node x and terminating at node y. Given anedge e, orig(e) determines the node from whiche originates, and term(e) determines the node atwhich e terminates. Let g1 = (V1, E1, α1, β1) andg2 = (V2, E2, α2, β2) be two graphs to be matched.Example 1

A graphical representation of two graphs beingmatched is shown in Fig. 1(b). These graphs havebeen derived from shape silhouettes, which areshown in Fig. 1(a). Such a process might be applied,for example, to the recognition of mechanical partsin a manufacturing line. The appearance of noise inany part of the image capture process may mean thatthe derived graphs, such as the ones shown in Fig. 1,do not match exactly. An inexact graph matching

Fig. 1 An example of two graphs to be matched. (a) Shapesilhouettes on which the graphs are based. (b) Derived graphs andnode numbering. (c) Node labels.

process is therefore required.Because the nodes in this example represent

locations in 2D space, they have been labeled with2D Cartesian coordinates (shown in Fig. 1(c)).Edges are not labeled in this example; they captureonly the structural information of the graphs.

The graphs shown in Fig. 1 are used throughoutthis paper to illustrate various concepts of theproposed algorithm. In particular, forming sequencesof correspondences and calculating the associatedcosts are described in Section 2.2. Transformingone graph into the other using graph edit operationsis described in Section 2.3. Finding the minimumcost match by tree search is presented in Section 2.4.Identifying collapsible partial matches is presented inSection 2.5. Grouping collapsible partial matches ispresented in Section 2.6. Collapsing for new graphsis presented in Section 2.7. The overall hierarchy ofmatching processes is described in Section 2.8. �

2.2 Inexact graph matching

Let a correspondence be an association between anode v1 ∈ V1 in g1, and a node v2 ∈ V2 in g2,(v1, v2) = V1 × V2. Let C be the set of all possiblecorrespondences between nodes in g1 and nodes ing2. One node in a correspondence c may be thenull node, Λ, which represents a node that is part ofneither g1 nor g2. A match, m, represents a sequenceof correspondences Cm = (c1, c2, · · · ), and a cost offorming its correspondences, mcost:

m = (Cm,mcost)In other words, a match is a binary relation

from a subset of V1, V1 ⊆ V1, to a subset ofV2, V2 ⊆ V2, along with the cost of performing

Page 4: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

294 Paul Morrison, Ju Jia Zou

such a mapping. A match’s cost is aligned with thedegree of similarity of the nodes that appear in thecorrespondences (differences between the labels ofcorresponding nodes), and the subgraphs implied bythe nodes that appear in the correspondences.

Note that the correspondences that appear in amatch are dependent on the edges in both graphs,and the allowed graph edit operations (describedin Section 2.3). While this research focuses oncorrespondences between graph nodes, it can easilybe extended to include correspondences betweengraph edges.

The method of forming sequences ofcorrespondences and calculating the associatedcosts is specific to the matching algorithm used.The method used in this research is discussed inthe following sections. An optimal inexact graphmatching algorithm is required to find the matchthat represents the optimal set of correspondences,where the optimal match is the match with theminimum global cost. A sub-optimal inexact graphmatching algorithm, however, is in general onlyrequired to find a local minimum-cost set ofcorrespondences. The local minimum may or maynot be close to the global minimum.Example 2

A sequence of correspondences for a match with apossible global minimum cost, for the graphs shownin Fig. 1(b), is {(1,1),(2,2),(3,3),(4,3),(5,4),(6,5)}. �

2.3 Graph edit operations

One of the oldest and most commonly usedmeasures of match optimality is the graph editdistance. Graph edit distance defines the similarityof two graphs in terms of the minimum set ofchanges required to transform one graph onto theother. Graph edit operations are used to obtainthe graph edit distance between two graphs. Asequence of graph edit operations is applied to onegraph to transform it into the other graph. Thissequence is implied by a match and its sequenceof correspondences. Three types of graph editoperations may be used to transform any graph intoanother. These are node/edge substitution, deletion,and insertion. We define a mapping function f of amatch m that places nodes of the two graphs intocorrespondence, determining Cm:

f : {V1,Λ} → {V2 ,Λ}

A node x ∈ V1 is substituted for node y ∈ V2if f(x) = y. A node x ∈ V1 is deleted from g1 iff(x) = Λ, and a node y ∈ V2 is inserted into g1 iff−1(y) = Λ.

2.4 Tree search algorithm

A tree (state space) search algorithm is oftenused to find the minimum cost match. The statespace consists of a number of states, whichrepresent matches, arranged in a tree structure. Thematch at the root of the state space contains nocorrespondences. The first layer of states representsmatches with one correspondence, the second layerwith two correspondences, and so on. Throughoutthis paper, the concepts of states and matches areused interchangeably in the context of the tree searchalgorithm. The tree search algorithm determines acost of placing any two nodes into correspondence,ccost. The cost associated with any given matchis equal to the sum of the costs of the match’scorrespondences.

A state (other than the root state) is producedby embedding a new correspondence in the state’sparent state. We define a function ϕ that determines,for a given state and its associated match, thepossible new correspondences that can be embeddedin the match to form new child states. The formthat ϕ takes is dependent on the specific tree searchmatching algorithm used.Example 3

Various definitions of ϕ have been proposed in theliterature. The following is an example that is similarto that used in Ref. [21]. Let Vm,i denote the set ofnodes in match m that belong to the graph gi, i ∈{1, 2}. Given m, ϕ returns correspondences betweennodes in g1 and nodes in g2 that are currentlynot matched (i.e., not in any correspondences ofmatch m), and are connected by edges to nodes thatare matched. In addition, correspondences betweenthese nodes and Λ can be made:ϕ(m) = {Vm,1 × Vm,2} ∪ {(x,Λ) : x ∈ Vm,1} ∪

{(Λ, y) : y ∈ Vm,2} (1)where Vm,i = {v : v ∈ (Vi−Vm,i), v ∈ {term(e) : e ∈Ei, orig(e) ∈ Vm,i}} defines those nodes that are notcurrently matched, but are connected to nodes thatare matched. Vm,i defines the nodes that can appearin any sub-match of match m. A sub-match of amatch m is a match that is produced by embedding

294

Page 5: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

Inexact graph matching using a hierarchy of matching processes 295

a new correspondence in m. �There are two considerations to take into account

when defining this function: (1) it affects whetherthe optimal match can be found, and (2) it affectsthe number of matches produced in the state space.In many cases, these considerations are traded offagainst each other. For example, a large numberof matches must often be produced in order tofind the optimal match. It is highly desirable thatthe number of matches produced (searched) shouldbe minimised, whilst guaranteeing that the optimalmatch will be found.

The first of the proposed changes in thisresearch is to vastly restrict the number ofcorrespondences returned by ϕ for any givenmatch. The implementation of ϕ used in this researchis based on that used by Eshera and Fu, givenin Eq. (1). We simplify this definition of ϕ bypreventing it from forming correspondences withΛ. This function therefore becomes:

ϕ(m) = {Vm,1 × Vm,2} (2)Modifying ϕ in this way has both the positive

effect of reducing the number of matches in thestate space, and the negative effect of reducing thealgorithm’s ability to find the optimal match. In thefollowing sections, we describe the methods that weuse to ensure that the optimal match is found aftersimplifying ϕ in this way.

It should be noted that other definitions of ϕ maybe used to implement the proposed algorithm. Thegoal is to simplify ϕ in such a way as to reduce thecomputational complexity of the matching process.The other requirement of ϕ is that “good” partialmatches should still be found by the tree searchalgorithm. The importance of this requirement willbecome apparent later.

2.5 Collapsibility criterion

During the tree search, partial matches will beproduced that have a low cost. Such matchesidentify correspondences between nodes that havesimilar attributes, and similar relationships betweencorresponding nodes, but are incomplete. Thesematches are likely to appear in a complete matchthat has a low overall cost. The intention of thecollapsibility criterion of the proposed method is toidentify such partial matches, which we loosely term“good” partial matches.

The proposed simplification of ϕ has the effectof preventing structural alterations during thematching process. Structural alterations are requiredto compensate for noise such as articulation andocclusion. The introduced collapsibility criterionreinstates the possibility of structural alterations,but only at certain points in the process. Thecollapsibility criterion allows structural alterationsto occur where “good” low cost partial matcheshave been identified. Allowing structural alterationsto occur at these points may be required to findthe lowest cost complete match. It is impossibleto provide an exact general definition of a “good”partial match, as it is application-specific. However,a “good” partial match should adhere to twoproperties:

1. Low cost. If the partial match has a low cost,then its correspondences are likely to appear ina complete match that has a low overall cost,which may be the global optimal match.

2. Minimum number of correspondences. Todetermine in a meaningful way that a partialmatch is likely to appear in a low costcomplete match, it should contain a numberof correspondences whose number is greaterthan or equal to a defined minimum number ofcorrespondences. This also serves to improvethe efficiency of the proposed method.

A mathematical example of the collapsibilitycriterion is given in Eq. (3). This example reflectsthe above properties of a “good” partial match, andgives an indication of the form that the collapsibilitycriterion may take. Results presented in Section4 demonstrate the practicality of this form ofcollapsibility criterion.Example 4

A “good” partial match for the graphs shownin Fig. 1(b) may contain the sequence ofcorrespondences {(1, 1), (2, 2), (3, 3)}. The cost offorming these correspondences would be low, yetthe match contains some minimum number ofcorrespondences (three). �

Identifying “good” partial matches is an importantaspect of the proposed algorithm. For reasonsthat will become apparent, we label such matchescollapsible during the creation of the state space.The condition that a match must satisfy in orderto be labeled collapsible is called the collapsibilitycriterion: collapsible = f(m, g1, g2).

Page 6: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

296 Paul Morrison, Ju Jia Zou

A partial match represents a match betweensubgraphs of the two graphs being matched. Thecollapsibility criterion therefore allows application-specific information at the subgraph scale to beapplied to the matching process.Example 5

The definition of the collapsibility criterion islargely application-specific. In this example, wedefine a “good” partial match as one that has acost that is substantially lower than the cost of anyof the match’s sub-matches, and has a number ofcorrespondences that is greater than some threshold:collapsible = (min(CSm)− cm > cthresh) and

(nm > nthresh) (3)where m is the match being tested for collapsibility,CSm is the set of costs of the sub-matchesof match m, cm is the cost of match m,nm is the number of correspondences in matchm, and cthresh and nthresh are the minimumthresholds for cost and number of correspondences,respectively. Using this collapsibility criterion,the match with correspondences {(1, 1), (2, 2), (3, 3)}between the two graphs shown in Fig. 1(b)will be considered collapsible, as it has a smallcost in comparison to the cost of its sub-matches (e.g., the match with correspondences{(1, 1), (2, 2), (3, 3), (4, 4)}), and it has the requirednumber of correspondences, assuming suitable valuesfor cthresh and nthresh. While the collapsibilitycriterion is application-specific, the example givenin Eq. (3) is general enough to apply to manyapplications. �

In Example 5, two thresholds are used. Whilethe collapsibility criterion is application-specific,it is likely that one or more thresholds will beneeded. It is unfortunate that thresholds andapplication-specific heuristics are often introducedwhen a graph matching algorithm is used in aspecific application. It will be interesting to seeif future inexact graph matchings can remove suchdependencies on thresholds.

2.6 Combining collapsible partial matches

After collapsible matches are identified, they arecombined to form groups of collapsible matches.This is done to realise “good” matches betweengroups of subgraphs. We define a grouping function,δ(Mc, g1, g2), which determines the possible groups

of matches given a set of collapsible matches, Mc.The simple grouping function that we use in thisresearch determines all possible combinations ofcollapsible matches such that no two matches in acombination have any common nodes:δ(Mc, g1, g2) = {comb(Mc, 2) ∪ comb(Mc, 3) ∪ · · ·

∪ comb(Mc, |Mc|) : Vm ∩ Vn = ∅,m 6= n, m, n ∈ comb(Mc, k),k = 2, 3, · · · , |Mc|}

where comb(Mc, i) denotes the set of all i-combinations of set Mc, and Vj denotes the set ofnodes in match j.

Note that this step may not be strictly necessary;the optimal match will be found without applyinga grouping function. However, this step allowsthe representation of application-specific informationacross matched subgraphs, which may be useful insome applications, and may serve to reduce therunning time of the proposed algorithm.

2.7 Collapsing for new graphs

For each group of collapsible matches, a collapseoperation is performed on g1 and g2, in order to formnew graphs, g′1 and g′2. The purpose of the collapseoperation is to preserve the fact that matches are“good”, so that the remainder of the nodes in thetwo graphs can be matched with this knowledge.As discussed earlier, collapsing also allows matchingto continue from different parts of the two graphs,allowing for structural alterations.

A collapse operation is performed for both graphsbeing matched, given a group of collapsible matches.In summary, for each graph, a new graph is producedthat is identical to the original graph with exceptionsregarding those nodes that appear in the group ofcollapsible matches. For each collapsible match inthe group, the matched nodes are replaced by asingle node in the new graph with a label thatis derived from the labels of the original nodes inthe match. Edges in the original graph are eitherdeleted, modified, or copied directly to the newgraph, depending on the originating and terminatingnodes of each edge. This process is defined formallyin the remainder of this section.

Let a new graph created by a collapse operationbe g′i = (V ′i , E′i, α′i, β′i), where i ∈ {1, 2} identifiesthe graph undergoing the collapse operation. Let Mdenote a group of collapsible matches, and Vi the

296

Page 7: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

Inexact graph matching using a hierarchy of matching processes 297

set of nodes in M that belong to the graph gi. Vi,j

denotes the set of nodes in graph gi, that belong toa particular collapsible match j. The set of nodes inthe new graph V ′i is defined as

V ′i = (Vi − Vi) ∪ {vj : 1 6 j 6 |M |}where vj are new nodes, each of which replaces a setof nodes of a collapsible match. Let V ′i,new = {vj :1 6 j 6 |M |} denote the new nodes in graph g′i. Theset of edges in the new graph is defined as follows:

E′i = (Ei − {exy : exy ∈ Ei, x, y ∈ Vi})∪{exy : x ∈ V ′i , y ∈ Vi → V ′i,new}∪{exy : x ∈ Vi → V ′i,new, y ∈ V ′i }∪{exy : x ∈ Vi,j , y ∈ Vi,k, j 6= k}

In essence, the edges in the new graph consist ofthose in the original graph with the exception ofthose between collapsed nodes, and with the additionof those that are from existing nodes to new nodes,from new nodes to existing nodes, and from newnodes to new nodes. Note that an edge exy fromthe original graph does not appear in the new graphif both its originating node, x, and its terminatingnode, y, are part of the same match being collapsed(x ∈ Vi,j and y ∈ Vi,k, where j = k).

The node labeling function for g′i is the same asfor gi, except where nodes are being collapsed andare replaced by new nodes. Where new nodes arebeing labeled, α′ will be application-specific, andwill typically assign labels that are derived fromthe labels of the nodes in the original graph thatis undergoing the collapse operation. The nodelabeling function for the new graph is defined asfollows:

α′i ={αi : Vi → L, if v /∈ Vi, v ∈ Vi

αnew : V ′i,new → Lnew, otherwisewhere Lnew is an alphabet of labels for new nodesand edges. αnew is essentially used to coalesce thelabels of the nodes being collapsed, and perhaps thelabels of the edges between those nodes, into a singlelabel for the new node.

The edge labeling function for g′i is the sameas for gi, except where one or both of an edge’soriginating and terminating nodes are new nodesformed through the application of the collapseoperation. When either of these nodes are newnodes, β′ will be application-specific, and willtypically assign labels that are derived from thelabels of the nodes that the edge connects or, if

applicable, the label(s) of the edge(s) that the newedge replaces. The edge labeling function for the newgraph is defined as follows:

β′i ={βi : Ei → L, if x, y /∈ Vi, exy ∈ Ei

βnew : E′i,new → Lnew, otherwisewhere E′i,new denotes the new edges in graph g′i.βnew is used to label those edges whose originatingnode, terminating node, or both originating andterminating nodes, are new nodes. Note that ifboth nodes x and y connected to edge exy are beingcollapsed and are from the same collapsible match(x ∈ Vi,j and y ∈ Vi,k, where j = k), β′i is notdefined (see E′i, above).Example 6

As mentioned previously, α′i and β′i areapplication-specific. In this example, we describea simple definition of α′ that might be used inmatching graphs with node labels derived frompoints in 2D space, such as those shown in Fig. 1.We wish to define αnew, the function that assignslabels to new nodes formed through applyingthe collapse operation. If we apply the collapseoperation to a set of nodes Vi,j in collapsible matchj, belonging to graph gi, we produce a new node v′jin the new graph g′i. The label of node v′j is equalto the average of the labels of the nodes in Vi,j :

αnew(j) = 1|Vi,j |

|Vi,j|∑k=1

α(vk)

where vk ∈ Vi,j . Here, the node labels are treatedas 2D vectors that, in the above equation, undergovector addition and scalar multiplication. As anexample, if nodes 1, 2, and 3 of graph g1 in Fig. 1(b)were to be collapsed, the label of the resulting nodein the new graph would be

(2, 16

3

). �

2.8 Hierarchy of matching processes

The overall algorithm, which uses the previouslydefined steps, will now be described with referenceto Fig. 2. The two graphs are first matched usinga matching process that has been simplified in themanner described in Section 2.4 (1 in Fig. 2). Thismatching process is referred to as Matching Process0 in Fig. 2. A portion of this matching process isshown in the figure, with the correspondences ofsome of the states that are produced, and the costsassociated with these states. During the matching

Page 8: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

298 Paul Morrison, Ju Jia Zou

Fig. 2 The proposed hierarchy of matching processes, applied to the example graphs of Fig. 1. n0 and n1 identify new nodes that are createdby applying the collapse operation.

process, matches are marked as collapsible if theysatisfy the collapsibility criterion. The state of onesuch collapsible match is State 3 of Matching Process0. A number of combinations of collapsible matchesare then formed. Each match in a combinationrepresents a subgraph of g1, and a subgraph of g2.Collapsing a combination is the process of derivingtwo new graphs from g1 and g2, denoted by g′1 andg′2. The process of collapsing State 3 of MatchingProcess 0 is shown as 2 in Fig. 2.

Each pair of new graphs, g′1 and g′2, producedby performing the collapse operation, is input toa new matching process (3 in Fig. 2). Thismatching process is referred to as Matching Process

1 for one of the possible pairs and MatchingProcess 2 for another possible pair in Fig. 2.Correspondences between new nodes resulting fromthe collapse operation are forced to appear in allmatches produced by the new matching process. Thenew matching process may produce new collapsiblematches based on the new graphs g′1 and g′2,such as the match associated with State 3 ofMatching Process 1, which are treated the sameway as in the initial matching process. That is,the matches are grouped, each group is collapsed,and each new pair of graphs is input to a newmatching process. This new matching process isreferred to as Matching Process 3 in Fig. 2. The

298

Page 9: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

Inexact graph matching using a hierarchy of matching processes 299

result of this recursive procedure is a hierarchyof matching processes. The overall hierarchicalmatching algorithm is summarised in Algorithm 1.Example 7

In Fig. 2, nodes are labeled with 2D Cartesiancoordinates. These labels are used to calculate thecost of forming each correspondence, in each matchin the state space. The initial correspondencesformed for the states in the first level of the statespace incur no cost. The cost of forming a newcorrespondence is determined by first measuringthe average direction from nodes previously putinto correspondence to the nodes in the newcorrespondence, for each graph. The cost is thenthe absolute difference between these two directions.Other cost functions can be defined in a similar way,for other forms of node and/or edge labels. Moredetails can be found in Ref. [22]. �

With each collapse operation, the original graphsare changed in some way, with the goal offinding a match that represents a more optimalset of correspondences. Just as each correspondenceformed by a matching process incurs a cost, acollapse operation must also have an associated costor penalty. This cost is added to the cost of all

Algorithm 1 : Hierarchical graph matching procedure1 11 11Input: two graphs to be matched: g1 and g2.Output: the lowest cost match between g1 and g2.

1 return MATCH(g1, g2)2 function MATCH(g1, g2)3 perform matching process on g1 and g24 lcm← lowest cost match from the matching

process5 identify collapsible matches from the matching

process (Section 2.5)6 form groups of collapsible matches (Section 2.6)7 foreach group of collapsible matches do8 apply the collapse operation, to form g′

1 andg′

2 (Section 2.7)9 sub lcm← MATCH(g′

1, g′2) (create a new

branch of the hierarchy using g′1 and g′

2, andobtain the lowest cost match from thesematching processes)

10 add cost of performing collapse to sub lcm

11 if sub lcm has a lower cost than lcm then12 lcm← sub lcm

13 end14 end15 return lcm

matches formed by any matching process that is theresult of applying the collapse operation (line 10 ofAlgorithm 1). For example, referring to Fig. 2, anymatches produced by Matching Process 1, or any ofits descendants in the hierarchy, will incur the costof performing the collapse operation at step 2 in thediagram. In this research, and in the example shownin Fig. 2, the cost of performing a collapse operationis set to a fixed value (0.1 in the diagram). However,it may easily be a function of the matches beingcollapsed. Once the hierarchy of matching processeshas been constructed, the optimal match is simplythe match with the lowest overall cost.

The proposed method differs from the traditionalinexact graph matching method in two main ways.Firstly, the proposed method simplifies ϕ, whichdrastically reduces the number of states thatare searched by a single matching process andconsequently removes the guarantee of optimality.Optimality is restored using the second aspect ofthe proposed method: the collapsibility criterion andhierarchical matching processes. The collapsibilitycriterion allows additional states to be searchedthat will possibly lead to the optimal solution. Theguarantee that the optimal solution is found amongstthe additional states is dependent on the application-specific collapsibility criterion; the criterion mustidentify those states that can lead to the optimalsolution. If this condition is satisfied, the optimalsolution is guaranteed to be found.

In general, the collapsibility criterion isapplication-specific. In designing the collapsibilitycriterion, it is useful to consider the sources ofnoise that are likely to appear in the chosenapplication, that will cause differences betweengraphs representing similar information. Thesesources of noise include articulation, occlusion,changes in viewpoint, and spurious noise that maycause additional nodes and/or edges in one or bothgraphs being matched.

It may seem that the proposed method dependsheavily on knowledge of the chosen application. Itis true that additional application-specific knowledgeis required to implement the proposed collapsibilitycriterion, and that this may effect the optimalityof the proposed method. In practice, however,the application will always be known. Manyknown optimisations, such as A*, utilise additional

Page 10: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

300 Paul Morrison, Ju Jia Zou

application-specific knowledge to achieve the samegoals (gains in efficiency). Relaxation-based methodsrequire additional information in order to determinegood initial assignments or probabilities. We havefound that a simple collapsibility criterion that hasthe properties discussed in Section 2.5 is toleranttoward noise, produces the optimal solution, andreduces the time taken to find the match.

3 Computational complexity analysis

3.1 Standard tree search

In this section, we analyse the computationalcomplexity of a typical inexact graph matchingalgorithm using tree search. We use the definitionof ϕ that is similar to Eshera and Fu’s, givenin Eq. (1). While this makes the following analysisspecific to one particular algorithm, this algorithmis typical of many. We aim only to capture thegeneral complexity of a class of algorithms sothat the benefits of the proposed method can bedemonstrated.

As before, let g1 and g2 be two graphs beingmatched. Let n1 and n2 be the number of nodes in g1and g2, respectively. For the purpose of this analysis,we assume that all edges are bi-directional, and thatboth graphs are fully-connected. We also assumethat all states in the tree must be searched to findthe optimal solution, which allows us to analyse thealgorithm as a depth-first tree search algorithm. Thecomputational complexity of depth-first tree searchalgorithm, as given by Ref. [23], is O(bm), where bis the branching factor (the maximum number ofsuccessors of any state in the tree), and m is themaximum depth of any state in the tree.

The upper bound on b occurs when a single pairof nodes has been put into correspondence (the firstlevel in the tree). A state with maximum depth m isproduced by applying a sequence of correspondencesbetween a node in g1 and Λ, followed by a sequenceof correspondences between Λ and a node in g2, orvice versa. These correspond to the second and thirdsets in the union of sets of ϕ, shown in Eq. (1).We can apply n1 or n2 of these correspondencesfollowed by n2 or n1 of these correspondences.The computational complexity of the standard treesearch algorithm, subject to the exact definition of ϕ

is therefore:O((n1n2 + 1)n1+n2

)(4)

3.2 The proposed algorithm

The proposed algorithm uses a matching processwith a lower computational complexity, but involvesthe use of multiple matching processes. As discussedin Section 2.8, the matching processes are arrangedin a hierarchy (tree). The overall algorithm maytherefore also be analysed as a depth-first tree searchalgorithm. The computational complexity of theproposed algorithm is O(bmh

h bmpp ), where bh and mh

are the branching factor and maximum depth forthe overall hierarchy, respectively, and bp and mpare the branching factor and maximum depth foran individual matching process within the hierarchy,respectively.

The branching factor for the overall hierarchy, bh,is the maximum number of sub-matching-processesproduced by a single matching process throughapplying the collapse operation. This has a one-to-one correspondence with the maximum number ofgroups of collapsible states produced by a matchingprocess, and therefore depends on the number ofcollapsible matches produced. The chosen definitionsof ϕ and δ (the grouping function) therefore playa role. Since the number of nodes in each graph,and therefore the number of states in each matchingprocess, reduce as the depth in the hierarchyincreases, the upper bound on the branching factorapplies to the first (root) matching process inthe hierarchy. bh can be derived by analysing thenumber of collapsible matches produced by the rootmatching process. The maximum depth of anymatching process in the hierarchy, mh, occurs whena sequence of collapses of single-match groups ofcollapsible matches is performed, and each matchbeing collapsed contains the minimum number ofnodes from each graph, nthresh.

We now turn our attention to the complexityof a single matching process within the hierarchyof matching processes. Recall that the proposedalgorithm aims to reduce the computationalcomplexity of a single matching process. If wesimplify ϕ as described in Section 2.4, that is,we remove the algorithm’s ability to performcorrespondences of nodes in either graph with Λ, thebranching factor of a single matching process caneasily be found. The maximum depth of a state in a

300

Page 11: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

Inexact graph matching using a hierarchy of matching processes 301

single matching process, mp, occurs when a sequenceof correspondences between a node in g1 and a nodein g2 are made, in the root matching process of thehierarchy. The overall computational complexity ofthe proposed algorithm, subject to choice of ϕ, is

O

((⌊min(n1, n2)nthresh

⌋!)⌊min(n1,n2)−1

nthresh−1

× ((n1 − 1)(n2 − 1))min(n1,n2))

(5)

In our analysis of this complexity, a number ofsimplifications have been made. Equation (5)provides an absolute worst-case complexity. In areal application, the branching factor and maximumdepth will typically be much smaller than the worstcase.

3.3 Computational complexity comparison

To more easily compare the computationalcomplexity of the two algorithms, we input variousvalues of n1 and n2 into Eq. (4) and Eq. (5) to giveupper bounds on the number of matches produced.The value of nthresh for the proposed method is alsovaried. These comparisons are shown in Table 1.

Observe that the worst-case computationalcomplexity of the proposed algorithm is muchsmaller than the standard tree search algorithm. Byincreasing nthresh, we can significantly reduce thenumber of matches produced (as there will be fewercollapsible matches in each state space). However,depending on the application, increasing nthresh mayreduce the algorithm’s ability to find the optimalmatch. In this research, a value of three has provedto be the most useful.

4 Experimental results

This section presents some results of the proposedalgorithm, in comparison with two previous optimalinexact graph matching algorithms that are based on

Table 1 Computational complexity comparison

Standard tree Proposed algorithm complexityn1, n2 search complexity nthresh = 2 nthresh = 3 nthresh = 4

5 1.41e14 1.68e7 1.05e6 1.05e66 6.58e18 1.90e12 9.77e8 2.44e87 6.10e23 3.66e15 6.27e11 7.84e108 1.02e29 1.52e23 2.66e14 1.33e149 2.81e34 1.98e27 2.33e19 7.21e1610 1.22e40 6.27e37 1.58e22 9.73e19

graph edit operations. The proposed method is thenextended to produce sub-optimal solutions, and iscompared with a recent sub-optimal shape matchingmethod. To distinguish the optimal and sub-optimalvariants of the proposed method, this section refersto them as the “optimal proposed method” and the“sub-optimal proposed method”, respectively.

The cost function used for the proposed methodand other optimal methods is described in Example7. A fixed cost of 0.1 is used for correspondencesbetween one node from either graph, and the nullnode. The collapsibility criterion used is describedin Example 5, with cthresh and nthresh set to 0.1 and3, respectively. All collapse operations incur a fixedcost of 0.1. All algorithms have been implementedusing the C++ programming language, and havebeen tested with a 2.52 GHz Intel CPU with 3.5 GBof RAM.4.1 Optimal inexact graph matching

In this section, the optimal proposed methodis compared with other optimal inexact graphmatching methods. The purpose of this section isto show the benefits of the proposed method whensearching for the optimal solution. The first of theprevious optimal algorithms chosen for comparisonsis due to Eshera and Fu [21]. This seminal algorithmutilises a state-space search method such as the onedescribed in Section 2.4, and performs graph editoperations of node insertion, node deletion, and nodesubstitution. This algorithm is typical of manyinexact graph matching algorithms in the literature.

The second previous optimal algorithm chosenfor comparison is that of Berretti et al. [24].Berretti et al. use an additional “node merging”operation, which essentially replaces a set ofconnected nodes with a new node. Whereasthe proposed algorithm implements a merging-likeoperation between matching processes, Berretti etal. perform their node merging operation as partof a single matching process. The addition of thisoperation therefore increases the complexity of asingle matching process. The suboptimal heuristicalso proposed by Berretti et al. is explored inSection 4.2.

These two previous algorithms have been chosenfor comparison to highlight the role that theemployed graph edit operations and the complexityof ϕ, play in determining the time taken to perform a

Page 12: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

302 Paul Morrison, Ju Jia Zou

match. We believe that, while the adverse influenceof the number of graph edit operations on matchtime may be apparent to those in the field, thisinfluence has received rather little analysis. Thefollowing results demonstrate this effect and theproposed algorithm’s ability to reduce it. Since allthree methods compared in this section are optimal,we compare match time only.4.1.1 Shape matchingThe three optimal methods are first tested usingshapes from the Kimia-216 shape database [25]. Thegraphs input to the three methods are derived fromthe skeletons of shape silhouettes. A sample ofthe shape silhouettes from the Kimia-216 databaseare shown in Fig. 3 with their derived graphsoverlaid. The shape skeletons were extracted usingthe method described by Morrison and Zou [26]. Thenodes of the derived graphs correspond to skeletonjunction and end points, and the edges of the derivedgraphs correspond to the skeleton branches betweenjunction and end points. The positions of the nodesin the plane are used for node attributes, and aretherefore used to calculate the costs of formingcorrespondences between nodes.

Figure 4 shows a histogram of the match timeusing the three optimal methods for shapes fromthe Kimia-216 database. The optimal proposedmethod performs almost all matches in under 10

Fig. 3 A sample of shapes from the Kimia-216 database and theirderived graphs.

Fig. 4 Histograms of match time for matching shapes from theKimia-216 database, using the optimal proposed method, Eshera andFu’s, and Berretti et al.’s.

seconds. While this is mostly true for the othermethods, a significant number of their matches arenot made in reasonable time (taking longer than60 seconds). We have also observed (though itis not shown) that the proposed method alwaystakes a shorter time than these previous methodsto perform any given match. Also, the methodof Eshera and Fu matches most combinations ofgraphs faster than that of Berretti et al., whointroduce an additional operation. This demostratesthe influence of the number of available graph editoperations on the complexity of the function thatdetermines each state’s successors. The proposedmethod’s simplification of a single matching processmore than offsets the additional time introduced byperforming multiple matching processes.4.1.2 Random graphsIn order to more fully test the proposed algorithm,“random” graphs were used to test the threeoptimal algorithms. The number of nodes andedges in each graph was controlled, while the nodelabels and configuration of edges were randomised.What resulted were graphs with known sizes, butvarying complexity. The purpose of these tests is todemonstrate the limitations of the three algorithmsin a controlled setting.

Figure 5 shows a plot of the average match timefor each of the algorithms, as the number of nodesin the graphs is increased. For a given number ofnodes and edges, five graphs were generated. Thenode attributes, and the originating and terminatingnodes of each edge, were randomised. The nodeattributes are, once again, 2D Cartesian coordinates,selected in the range 0.0 to 5.0 in the x- and

Fig. 5 Match time vs. number of nodes for random graphs. Runstaking longer than 60 seconds are aborted.

302

Page 13: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

Inexact graph matching using a hierarchy of matching processes 303

y-directions. Each graph is matched only withother graphs with the same number of nodes andedges. Results of matching graphs with differentcharacteristics are presented in Section 4.1.1.

All three algorithms are able to complete matchingin a reasonable time when the number of nodes in thegraphs is small (four or fewer nodes). As the numberof nodes is increased, there is a point at which anyalgorithm becomes unable to complete in under 60seconds. The optimal proposed method is faster thanthe two previous methods, but fails to perform thematch in a reasonable time with some number ofnodes (seven nodes). This is due to the inherentcomplexity of the inexact graph matching process,which is retained by the proposed algorithm.

Note that we have presented here match time forthree algorithms implemented without the use ofheuristics or estimations to guide the search; all threealgorithms perform an exhaustive search of the statespace. This has been done in order to more clearlydemonstrate the effect that available operations hason match time, and the advantages of the proposedmethod.

4.2 Sub-optimal inexact graph matching

In the previous section, the proposed methodwas compared with two other optimal methodsthat are based on tree search and graph editoperations. While the proposed method shows a clearadvantage in terms of match time, it cannot escapethe underlying high computational complexity ofoptimal matching. In this section, we compare theoptimal proposed method with two other matchingmethods that are sub-optimal. The first sub-optimal method is the (optimal) proposed methodaugmented with a simple technique proposed byBerretti et al. [24] for improving match speed whileproducing only a sub-optimal solution. We namethis method the “sub-optimal proposed method”, asopposed to the “optimal proposed method”. Thesecond sub-optimal method is the method of Bai andLatecki [5]. Bai and Latecki also use shape skeletonsto match shapes, but their technique is somewhatdifferent. Their approach is to compare shape widthssampled from the shortest “skeleton paths” betweenpairs of end points, and they do not considerskeleton junction points or the structure of theskeleton explicitly. Despite the different approachesof these methods, comparing their results allows us

to evaluate the proposed method in the context ofshape matching, and the state of the art in thisfield. We set all parameters of Bai and Lateckito those values given in Ref. [5]. All methods arecompared in terms of accuracy using the Kimia-216shape database. The sub-optimal algorithms testedalways produce a match quickly (usually in less thanone second), so a comparison in terms of match timeis not given.

Out of 2376 matches (the 11 closest matchesfor each of the 216 shapes), the optimal proposedmethod matches 767 shapes from the same categoryas the query shape. The retrieval rate of the optimalproposed method for the Kimia-216 database istherefore 32.3%. Using similar calculations, the sub-optimal proposed method and the method of Baiand Latecki have retrieval rates of 31.8% and 34.8%,respectively. While these retrieval rates do notappear promising, it should be noted that they applyto the current use of the proposed method in theshape matching application. The performance of thethree methods for the shape matching applicationis explored in more detail in the remainder of thissection, and various improvements are proposed.

Figure 6 shows a selection of matches madeusing the optimal proposed method. The optimalproposed method performs reasonably well for mostshapes, considering it utilises skeleton informationonly. Contour information, while used to producethe skeleton, is not used by the proposed matchingalgorithm directly. In addition, the implementationof the proposed method uses the simple cost functiondescribed in Example 7 that is based purely onrelative angle differences. The results shown inFig. 6 should therefore be evaluated in terms ofthe underlying graphs only. For the shape matching

Fig. 6 Selected results of the proposed method on the Kimia-216database.

Page 14: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

304 Paul Morrison, Ju Jia Zou

application, the proposed method can potentially beimproved with a more sophisticated cost functionthat utilises more information.

The proposed method can easily be modified toproduce a sub-optimal solution in much shortertime, if required. Berretti et al. [24] have chosen toexplore only the best cost match of the possible nextmatches at each iteration. This means that thereare many branches of the state space that are leftunsearched, one of which may contain the globaloptimal match. To demonstrate that the proposedmethod can be extended to provide a sub-optimalsolution if required, it is modified using the approachtaken in Ref. [24]. Sample results from this methodare shown in Fig. 7. The results from the sub-optimal proposed method deviate slightly from thoseof the optimal proposed method. Overall, the sub-optimal proposed method is not as accurate as theoptimal proposed method, in terms of matching theunderlying graphs. However, in most cases, thedifference is not significant.

As mentioned previously, the implementation ofthe proposed method does not consider shapecontour information explicitly during the matchingprocess. This clearly affects the results, from theperspective of matching shape silhouettes. Bai andLatecki have recently proposed a shape matchingmethod that uses shape skeletons combined withshape widths at equidistant locations along skeletonpaths between endpoints. Some results from testingthis method using the same shapes from the Kimia-216 database are shown in Fig. 8.

For the shape silhouette matching task, themethod of Bai and Latecki outperforms both theoptimal and sub-optimal proposed methods in most

Fig. 7 Selected results of the sub-optimal proposed method,modified using a “best-only” state space search method, on the Kimia-216 database.

Fig. 8 Selected results of the shape matching method of Bai andLatecki [5] on the Kimia-216 database.

cases. The overall retrieval rate of Bai and Lateckiis also slightly higher than the proposed method.We believe that this is because they make use ofimportant shape contour information directly in thematching process. We note that we could not obtainthe outstanding results given by Bai and Latecki [5].This is most likely due to the use of a differentskeletonization method. Shape matching accuracy isheavily dependent on both the shape representation(in this case, the skeleton) and the shape matchingmethod itself. To achieve high accuracy, the shaperepresentation and matching method must be closelyaligned.

There are some cases where the method of Bai andLatecki clearly does not produce accurate results.Bai and Latecki do not use structural informationcontained in the skeleton. This appears to put theirmethod at a disadvantage for some shapes, suchas the second query shape of Fig. 8. Here, theskeleton captures the structure of the underlyingshape but, because the shape does not consistprimarily of ribbon-like parts, shape widths alongskeleton branches may not be a good representationof contour information. The skeleton representationis known to be of most utility when representingribbon-like shapes. The fourth query shape in Fig. 8,on the other hand, consists entirely of ribbon-likeparts and the method of Bai and Latecki performsextremely well. This highlights that, while placingmore emphasis on shape structure definitely addscomplexity, it is required for robust shape matching.

Finally, we illustrate the applicability of theproposed method to the text character recognitionproblem. From a matching perspective, textcharacters are somewhat different to the shapescontained in the Kimia-216 database. Because text

304

Page 15: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

Inexact graph matching using a hierarchy of matching processes 305

characters consist mainly of strokes of comparativelyuniform width, structural information must play akey role in matching. Furthermore, text charactersoften contain loops, which presents problems tomany matching methods. Figure 9 shows someresults of matching in a database of eight characters(“a” to “h”) from six different fonts.

While the proposed inexact graph matchingalgorithm can match graphs containing loops orcycles, the current application of the algorithm tothe shape matching problem does not capture shapeloops effectively (observe results for the “b” characterin Fig. 9). The graphs derived from shape skeletonscontain nodes placed on skeleton junctions and endpoints only. To capture loops, nodes must be placedon skeleton branches (e.g., at locations of highcurvature), or graph edges must somehow representpositions along branches. The proposed inexactgraph matching algorithm can easily accommodatesuch additional information. Loops appear to play alarge role in differentiating text characters, and aretherefore a very important aspect of the characterrecognition problem.

4.3 Discussion

While sub-optimal methods outperform theproposed method and other optimal methodsin terms of match time, some sacrifice in quality ismade. The sub-optimal proposed method performswell in most cases, and shape matching retrievalrates are similar. However, when results are analysedin terms of the graph information only, the sub-optimal proposed method is not as accurate as theoptimal proposed method.

The proposed method matches nodes in a waythat corresponds with a human’s perception of

Fig. 9 Selected results of the proposed method on a text characterdatabase.

the graph structures. However, in order to matchshape silhouettes in a more meaningful way, contourinformation must be used directly in the matchingprocess. The simple cost function used in thisapplication of the proposed method does not considershape contour information. The absence of thisinformation is reflected in the results. In futureresearch, the cost function used here will be extendedto include contour information.

While contour information appears to beimportant for matching shapes from the Kimia-216 database, structural information is moreimportant for text character recognition. Therelative importance of contour and structural shapeinformation varies depending on the type of shapesbeing matched. A balance between the two is clearlyrequired for generic shape matching. The proposedmethod’s performance in text character matchingcan further be improved by using additionalskeleton-based information (such as nodes in areasof high curvature) to aid in differentiating charactersthat contain loops.

The proposed method, being based on earlyattributed relational graph (ARG) matchingmethods, can easily be extended in these directionswithout any change to the underlying algorithm.These changes will improve the proposed method’sperformance in the application of shape matching.

Most recent work on inexact graph matching hasattempted to produce good sub-optimal solutions.This paper takes a very different approach, andattempts to reduce the complexity of finding theoptimal solution. Such an approach is, of course,beneficial only if the optimal solution is requiredor presents a significant improvement over theexpected sub-optimal solution. The results shownin this section demonstrate both the advantages anddisadvantages of the proposed approach: there is alimit to the size of the graphs that can be matched,but it produces better results than a sub-optimalmethod in many cases. We believe that optimalsolutions are of high interest to the community,and that finding optimal solutions will become morefeasible in the near future.

It may be argued that the inexact graph matchingproblem itself implies uncertainty, and therefore anoptimal solution is not necessary. However, we arguethat the uncertainty inherent in the problem itself

Page 16: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

306 Paul Morrison, Ju Jia Zou

should be separated from the uncertainty incurredby the matching algorithm. The ultimate goal ofa matching algorithm is to determine how similartwo graphs are. Erroneous or sub-optimal resultsmay have a detrimental effect on the decisions madeusing such similarity measures. Many applications,such as handwriting recognition or optical characterrecognition (OCR), rely on accurate similaritymeasures in order to match shapes correctly.

We have already mentioned that reducingthe complexity of computationally-intensive graph-theoretic algorithms is gaining renewed interest.Also, with the increasing use of parallel processingtechniques, computationally-intensive algorithmsdesigned for such parallelism will make manypreviously-intractable methods useful. The proposedmethod is amenable to such techniques, which willbe used to develop the proposed method in futureresearch.

5 Conclusions

In this paper, we have proposed a method ofreducing the computational complexity of theinexact matching process. The proposed methodis able to produce the optimal solution, subjectto an application-specific criterion that operates atthe subgraph level. Experimental results and acomparison of the computational complexity of theproposed algorithm with a previous algorithm thatis based on tree search demonstrate that our aim hasbeen achieved.

We believe that this work contributes significantlyto the community, by proposing an approach to theinexact graph matching problem that is substantiallydifferent to the existing literature, demonstrating themethod’s application to a common use of inexactgraph matching, and demonstrating the method’spotential use in other applications by including anestimation that allows for sub-optimal matches. Forthe application of shape matching, various potentialimprovements to the implementation of the proposedmethod are discussed.

While the implementation presented in this paperis based on the tree search algorithm of a particularprevious study, we believe that the technique maybe expanded to other methods of performing inexactgraph matching. These ideas may even be extendedto inexact graph matching algorithms that are not

based on tree search and graph edit operations. Thisis an area for future research. By breaking the overallmatching process down into a number of smallermatching processes, the proposed method can easilybe implemented with parallel processing in mind.

Results show that the proposed method is ableto match graphs with more nodes in a reasonableamount of time.

Acknowledgements

The authors wish to express their gratitude toassociate professor Weidong (Tom) Cai at theUniversity of Sydney in Australia for his advice andhelp in relation to the publication of this article.

Open Access This article is distributed under theterms of the Creative Commons Attribution License whichpermits any use, distribution, and reproduction in anymedium, provided the original author(s) and the source arecredited.

References

[1] Conte, D.; Foggia, P.; Sansone, C.; Vento, M. Howand why pattern recognition and computer visionapplications use graphs. In: Studies in ComputationalIntelligence, Vol. 52. Kandel, A.; Bunke, H.; Last, M.Eds. Springer-Verlag Berlin Heidelberg, 85–135, 2007.

[2] Hu, S.-M.; Zhang, F.-L.; Wang, M.; Martin, R.R.; Wang, J. PatchNet: A patch-based imagerepresentation for interactive library-driven imageediting. ACM Transactions on Graphics Vol. 32, No.6, Article No. 196, 2013.

[3] Vento, M.; Foggia, P. Graph matching techniquesfor computer vision. In: Graph-Based Methods inComputer Vision: Developments and Applications.Bai, X.; Cheng, J.; Hancock, E. Eds. IGI Global, 1–41,2012.

[4] Wang, M.; Lai, Y.-K.; Liang, Y.; Martin, R. R.; Hu,S.-M. BiggerPicture: Data-driven image extrapolationusing graph matching. ACM Transactions on GraphicsVol. 33, No. 6, Article No. 173, 2014.

[5] Bai, X.; Latecki, L. J. Path similarity skeleton graphmatching. IEEE Transactions on Pattern Analysis andMachine Intelligence Vol. 30, No. 7, 1282–1292, 2008.

[6] Tsai, W.-H.; Fu, K.-S. Error-correcting isomorphismsof attributed relational graphs for pattern analysis.IEEE Transactions on Systems, Man and CyberneticsVol. 9, No. 12, 757–768, 1979.

[7] Tsai W.-H.; Fu, K.-S. Subgraph error-correctingisomorphisms for syntactic pattern recognition. IEEETransactions on Systems, Man and Cybernetics Vol.13, No. 1, 48–62, 1983.

306

Page 17: Inexact graph matching using a hierarchy of …...Inexact graph matching using a hierarchy of matching processes 293 estimations, but rather allows optimizations through the use of

Inexact graph matching using a hierarchy of matching processes 307

[8] Berretti, S.; Bimbo, A. D.; Vicario, E. Efficientmatching and indexing of graph models in content-based retrieval. IEEE Transactions on PatternAnalysis and Machine Intelligence Vol. 23, No. 10,1089–1105, 2001.

[9] Nilsson, N. J. Problem-solving Methods in ArtificialIntelligence. McGraw-Hill Pub. Co., 1971.

[10] Christmas, W. J.; Kittler, J.; Petrou, M. Structuralmatching in computer vision using probabilisticrelaxation. IEEE Transactions on Pattern Analysisand Machine Intelligence Vol. 17, No. 8, 749–764,1995.

[11] Gold, S.; Rangarajan, A. A graduated assignmentalgorithm for graph matching. IEEE Transactions onPattern Analysis and Machine Intelligence Vol. 18, No.4, 377–388, 1996.

[12] Conte, D.; Foggia, P.; Sansone, C.; Vento, M.Thirty years of graph matching in pattern recognition.International Journal of Pattern Recognition andArtificial Intelligence Vol. 18, No. 3, 265–298, 2004.

[13] Foggia, P.; Percannella, G.; Vento, M. Graph matchingand learning in pattern recognition in the last 10years. International Journal of Pattern Recognitionand Artificial Intelligence Vol. 28, No. 1, 1450001,2014.

[14] Livi, L.; Rizzi, A. The graph matching problem.Pattern Analysis and Applications Vol. 16, No. 3, 253–283, 2013.

[15] Gregory, L.; Kittler, J. Using graph search techniquesfor contextual colour retrieval. In: Lecture Notes inComputer Science, Vol. 2396. Caelli, T.; Amin, A.;Duin, R. P. W.; de Ridder, D.; Kamel, M. Eds.Springer-Verlag Berlin Heidelberg, 186–194, 2002.

[16] Eppstein, D. Quasiconvex analysis of multivariaterecurrence equations for backtracking algorithms.ACM Transactions on Algorithms Vol. 2, No. 4, 492–509, 2006.

[17] Fomin, F. V.; Grandoni, F.; Kratsch, D. Measure andconquer: Domination—A case study. In: Lecture Notesin Computer Science, Vol. 3580. Caires, L.; Italiano,G. F.; Monteiro, L.; Palamidessi, C.; Yung, M. Eds.Springer-Verlag Berlin Heidelberg, 191–203, 2005.

[18] Fomin, F. V.; Grandoni, F.; Kratsch, D.; Lokshtanov,D.; Saurabh, S. Computing optimal steiner trees inpolynomial space. Algorithmica Vol. 65, No. 3, 584–604, 2013.

[19] Van Rooij, J. M. M.; Bodlaender, H. L. Exactalgorithms for edge domination. Algorithmica Vol. 64,No. 4, 535–563, 2012.

[20] Woeginger, G. J. Exact algorithms for NP-hardproblems: A survey. In: Lecture Notes in ComputerScience, Vol. 2570. Junger, M.; Reinelt, G.; Rinaldi,G. Eds. Springer-Verlag Berlin Heidelberg, 185–208,2003.

[21] Eshera, M. A.; Fu, K.-S. A graph distance measure forimage analysis. IEEE Transactions on Systems, Manand Cybernetics Vol. 14, No. 3, 398–408, 1984.

[22] Morrison, P. Shape matching based on skeletonisationand inexact graph matching. Ph.D. thesis. WesternSydney University, 2011.

[23] Russell, S.; Norvig, P. Artificial Intelligence: AModern Approach. Prentice-Hall, 1995.

[24] Berretti, S.; Bimbo, A. D.; Pala, P. A graph editdistance based on node merging. In: Lecture Notes inComputer Science, Vol. 3115. Enser, P.; Kompatsiaris,Y.; O’Connor, N. E.; Smeaton, A. F.; Smeulders, A.W. M. Eds. Springer-Verlag Berlin Heidelberg, 464–472, 2004.

[25] Sebastian, T. B.; Klein, P. N.; Kimia, B. B.Recognition of shapes by editing shock graphs. In:Proceedings of the 8th IEEE International Conferenceon Computer Vision, Vol. 1, 755–762, 2001.

[26] Morrison, P.; Zou, J. J. Triangle refinement in aconstrained Delaunay triangulation skeleton. PatternRecognition Vol. 40, No. 10, 2754–2765, 2007.

Paul Morrison completed his Ph.D.degree in 2011 at the Western SydneyUniversity, Australia, whilst studyingtopics in image processing and shaperecognition. He currently pursuesresearch and development at CiSRAin Sydney, Australia, and is a memberof the Institute of Electrical and

Electronics Engineers (IEEE).

Ju Jia Zou received his B.S. andM.S. degrees in radio-electronicsfrom Zhongshan University (alsoknown as Sun Yat-Sen University)in Guangzhou, China, in 1985 and1988, respectively, and Ph.D. degreein electrical engineering from theUniversity of Sydney, Australia, in

2001. Currently, he is a senior lecturer with School ofComputing, Engineering and Mathematics at WesternSydney University, Australia. He was a research associateand then an Australian postdoctoral fellow at the Universityof Sydney from 2000 to 2003. His research interests includeimage processing, pattern recognition, computer vision,and their applications. He has been a chief investigator fora number of projects funded by the Australian ResearchCouncil. He is a member of the Institute of Electrical andElectronics Engineers (IEEE).

Other papers from this open access journal are available freeof charge from http://www.springer.com/journal/41095.To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.