including the size of regions in image segmentation by region-based graph

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 2, FEBRUARY 2014 635

Including the Size of Regions in ImageSegmentation by Region-Based Graph

Alireza Rezvanifar and Mohammadali Khosravifard, Member, IEEE

Abstract— Applying a fast over-segmentation algorithm toimage and working on a region-based graph (instead of the pixel-based graph) is an efficient approach to reduce the computationalcomplexity of graph-based image segmentation methods. Never-theless, some undesirable effects may arise if the conventional costfunctions, such as Ncut, AverageCut, and MinCut, are employedfor partitioning the region-based graph. This is because thesecost functions are generally tailored to pixel-based graphs. Inorder to resolve this problem, we first introduce a new classof cost functions (containing Ncut and AverageCut) for graphpartitioning whose corresponding suboptimal solution can beefficiently computed by solving a generalized eigenvalue problem.Then, among these cost functions, we propose one that considersthe size of regions in the partitioning procedure. By simulation,the performance of the proposed cost function is quantitativelycompared with that of the Ncut and AverageCut.

Index Terms— Graph-based image segmentation, Ncut,AverageCut, spectral clustering, region-based graph, mean-shiftalgorithm.

I. INTRODUCTION

IMAGE segmentation is the process of partitioning an imageinto disjoint homogeneous regions so that all pixels in a

region are similar with respect to some characteristics suchas color, intensity, or texture, while adjacent regions aresignificantly different with respect to the same characteristics.Segmentation is one of the essential low-level vision opera-tions with a fundamental role in the final result of higher-level operations such as object recognition and tracking, imageretrieval, face detection, etc. Clearly, color images containmuch more information than gray-level ones, which can beused to improve the quality of segmentation. Although thisimprovement is obtained at the expense of computationalcomplexity, it is no longer a major problem with the recentincrease in processing power. Accordingly, in this paper, weconsider color image segmentation problem.

Thus far, numerous segmentation algorithms have beenproposed in the literature (See [1]– [4] for more reviews).

Manuscript received November 26, 2012; revised August 12, 2013; acceptedOctober 17, 2013. Date of publication November 7, 2013; date of currentversion December 17, 2013. This work was supported by a grant fromIran Research Institute for ICT (ITRC). The associate editor coordinat-ing the review of this manuscript and approving it for publication wasProf. Ton Kalker.

The authors are with the Department of Electrical and Computer Engineer-ing, Isfahan University of Technology, Isfahan 84156-83111, Iran (e-mail:[email protected]; [email protected]).

This paper has supplementary downloadable material available athttp://ieeexplore.ieee.org, provided by the author. This supplementary materialconsists of 125 JPEG images which represent the results for 25 selectedimages.

The total size of the files is 2.63 MB. Contact [email protected] forfurther questions about this work.

Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TIP.2013.2289984

Generally, these algorithms can be categorized into thefollowing three categories: 1) feature-space based [5], [6], 2)image-domain based [7]–[9], and 3) physics based [10], [11].

The main approach of the methods in the first category is totransfer the image information into a feature space in whichan appropriate clustering algorithm can be applied. Thesemethods generally ignore the spatial relationship between theimage pixels. This may result in homogeneous regions whichare not spatially connected. Methods in the second categoryreduce this effect by utilizing the spatial information of pixels.In the third category, the effect of highlights and shadows isreduced by considering the physical interaction of light withthe colored materials. Some methods in these categories arereviewed in [2] and [3].

A. Graph-Based Image Segmentation

Generally, efficient clustering algorithms (including imagesegmentation methods) should follow the general commonsense principle that the objects within a cluster show a highdegree of similarity, while those in different clusters exhibitlower affinity. One of the attractive approaches that can achievethis property is the one based on the graph theory. It has awide variety of applications in different fields of clustering likeimage segmentation [8], [9], [12]– [17], image categorization[18], video scene clustering [19], spatiotemporal segmentation[20], etc.

In the context of image segmentation, a graph whichdescribes the image in some sense is partitioned into a set ofconnected components corresponding to image segments. Themethods following this approach lie in the second categorymentioned above. They can properly consider both featureand spatial information in the constructed graph. Extractingthe global impression of the image and involving the infor-mation of all pixels in the segmentation process are the otheradvantages of the graph-based methods [8].

In this paper, we focus on the graph-based image segmen-tation. The general framework of this approach is as follows.First, a weighted graph G = (V , E, W ) is constructed withnodes corresponding to pixels/regions of the image. Here,V = {1, 2, . . . , n} and E denotes the set of all nodes andedges, respectively. Also, W = [wi j ]n×n is a weight matrix,where wi j represents the similarity between nodes i and j . Theweights are usually defined based on the brightness, color ortexture features, as well as the spatial characteristics of thecorresponding pixels/regions. In this way, the image segmen-tation problem can be reformulated as a graph partitioningproblem. That is, the graph is partitioned into K components(A1, A2, . . . , AK ), where Ai represents the i -th segment ofthe image. To do so, the quality of each potential partitioningmust be evaluated by a predefined cost function.

1057-7149 © 2013 IEEE

636 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 2, FEBRUARY 2014

For partitioning a graph into K partitions (K > 2) basedon a given cost function, two strategies can be considered [8]:1) recursive two-way and 2) simultaneous K-way. With theformer strategy, in each recursion of the algorithm one ofthe obtained components is partitioned into two components,until some termination criterion is met. Obviously, with thisstrategy, large and salient segments should be separated inthe initial recursions and details come afterward [8]. On theother hand, with the second strategy, all K required partitionsare obtained simultaneously [12]. In doing so, the number ofdesirable partitions should be determined a priori. In this paper,the first strategy is used for partitioning the graph. Thus, themain problem is to devise an appropriate cost function forevaluating any bipartitioning (A, A).

According to the aforementioned general common senseprinciple, such a cost function should lead to clusters A andA with high inter-cluster and/or low intra-cluster similarities.Moreover, it is preferred that with the devised cost function,the optimal or suboptimal solution can be found in a reason-able time with low computational complexity.

The first cost function that has been employed in the imagesegmentation is the cut value [13], which is defined by

cut(A, A) =∑

i∈A , j∈A

wi j .

The main advantage of this approach, known as “MinCut”,is the existence of a polynomial-time algorithm for derivingthe optimal solution. However, this cost function has a biastowards short boundaries and hence, small sets of isolatednodes are more likely to be separated in the initial recursionsof segmentation. Several cost functions were proposed in theliterature to reduce this effect [8], [14], [15], [16], [21]. Oneof the most popular ones is the “Normalized Cut” (or “Ncut”)criterion [8], which is defined by

Ncut(A, A) = cut(A, A)

assoc(A, V )+ cut(A, A)

assoc(A, V ), (1)

whereassoc(A, V ) =

∑

i∈A , j∈V

wi j .

This approach tends to generate balanced partitions whilefollowing the general common sense principle.

Each partitioning (A, A) is associated with an indicatorvector x = (x1, x2, . . . , xn) defined by

xi ={

+1 i ∈ A

−1 i ∈ A, (2)

where n denotes |A ∪ A|. It has been proven that minimizingthe Ncut criterion is an NP-Complete problem. However, itis shown in [8] that if x is relaxed to take real values, thenthe optimum indicator vector would be the eigenvector corre-sponding to the second smallest eigenvalue of the generalizedeigenvalue problem

(D − W )x = λDx. (3)

In (3), the Degree matrix of the graph D, is a diagonalmatrix with entries dii = ∑

j∈V wi j on the main diagonal.

Clearly, the real vector obtained by solving (3) must bediscretized before partitioning the image. To do so, somedifferent ways are proposed in [8].

B. Region-Based Graph

Although some efficient algorithms have been proposed forsolving (3), they suffer from high computational complexitywhen the number of nodes is large. In particular, if each nodeof the graph represents a pixel of the image, i.e., we usepixel-based graph, solving the problem is not feasible even forimages of moderate size. Several methods have been proposedin the literature for resolving this problem [9], [22]–[25]. Oneof the efficient approaches is to construct a graph whose nodesrepresent small regions of the image. With this method, a setof over-segmented regions are initially generated by a fastalgorithm (such as Mean-shift [6], [26], SLIC [27], watershed[9], etc.), and then a graph structure is used to representthe relationship between these regions [9], [22], [28]. In thenew graph, which is called region-based graph, each noderepresents a region. In this way, the number of nodes canbe considerably reduced and graph-based algorithms such asNcut can be implemented with an acceptable computationalcomplexity.

To consider the similarity between different pixels in thepixel-based graph, each node must be adjacent to many nodesin its neighborhood. Since such a similarity between pixels hasbeen considered by the primary over-segmentation algorithm,the nodes in the region-based graph need to be adjacent withonly few other nodes representing their adjacent regions [22].

The weights of the graph can be defined by the conventionalexponential weight function [8]

wi j =

⎧⎪⎨

⎪⎩

e−‖Fi −Fj‖22/dI , if i and j are adjacent

0, otherwise

, (4)

where Fi and Fj reflect the intensity and/or color informationof the i -th and j -th regions and ‖.‖2 denotes the vector normoperator. Also, the parameter dI , which can strongly influencethe final result of the segmentation, is a positive scaling factorthat determines the sensitivity of wi j to the color differencebetween nodes i and j .

C. Problem

Generally, in the region-based graph, a node may be associ-ated with an extremely large or small region of the image [Seeregions 1 and 5 in Fig. 2.(a)]. However, these nodes are of thesame importance when applying the Ncut criterion. Therefore,the tendency of Ncut to generate balanced partitions preventsseparating isolated nodes corresponding to large regions.

For instance, consider the grayscale image of Fig. 2.(a). Thebrightness of this image is 0, 40, 80, 120 and 160 in regions1-5, respectively. Clearly, it is desired to identify region 5 ofFig. 2.(a) in the first recursion of the bipartitioning algorithm.But, for a wide range of dI (10 ≤ dI ≤ 2000), applying Ncuton the region-based graph of Fig. 2.(b) leads to one of thesegmentations shown in Fig. 2.(c) and 2.(d).

REZVANIFAR AND KHOSRAVIFARD: INCLUDING THE SIZE OF REGIONS IN IMAGE SEGMENTATION 637

Fig. 1. (a) First row: original image, output of Mean-shift (254 regions) and SLIC (176 regions). (b) First row: original image, output of Mean-shift(202 regions) and SLIC (180 regions). In (a) and (b), the third (resp. second) row represents segmentation with the Ncut (resp. proposed cost function) appliedto the output of SLIC (resp. Mean-shift algorithm) after 5, 10 and 15 recursions.

Fig. 2. (a) Original image, (b) region-based graph, (c) and (d) two possibleresults by using Ncut.

This example shows that generating balanced partitions andpreventing separation of isolated nodes, which was the mainadvantage of the Ncut for pixel-based graphs, turns to be adisadvantage for region-based graphs. Note that MinCut is nota proper criterion as well. This is because it may separateisolated nodes corresponding to small regions in the initialsteps of the partitioning process.

To address this, one way is to use an over-segmentationalgorithm like SLIC [27] or Turbopixels [29] which generatesregions (superpixels) of similar sizes. In this case, there is nosignificant difference between the size of regions and hence,Ncut can be used for partitioning the resulting region-basedgraph. But the problem is that the size (resp. number) ofsuperpixels should be made small (resp. large) enough so thatall desired objects/segments of the image would be detectable.In this way, large smooth regions will be partitioned into somany superpixels and consequently, some false boundariesare generated which affect the final output of the imagesegmentation process. This effect is illustrated in Fig. 1.

The third row of this figure shows the result of applying Ncuton the superpixels for 5, 10 and 15 recursions, respectively.

Following [22], in this paper, we use Mean-shift algorithmas the primary over-segmentation algorithm to construct theregion-based graph. Although Mean-shift algorithm is slowerthan SLIC, it is able to generate irregularly shaped regions ofnon-uniform sizes and considerably reduce false boundaries.Also, it preserves the desirable discontinuity characteristics ofthe image. The second row in Fig. 1 illustrates the result ofsegmentation process after 5, 10 and 15 recursions when thecost function proposed later in this paper is applied to theoutput of Mean-shift algorithm.

In order to separate large and salient regions of the imagein the initial recursions of partitioning the region-based graph,the size of regions should be somehow considered in the costfunction. In this paper, we characterized those cost functionswhose corresponding suboptimal solution could be obtainedby an approach very similar to Ncut. Then we focused on oneof these cost functions which took the size of regions intoaccount in inter-cluster similarity. By using this cost function,in addition to spatial and feature space information, the sizeof regions influences the segmentation procedure.

This paper is organized as follows. In the next section, arelated work on resolving the problem of using Ncut for theregion-based graphs [22] is investigated. In Section III, weintroduce a new family of cost functions and propose oneof them for partitioning the region-based graphs. Simulationresults are presented in Section IV to compare the performanceof the proposed cost function with other well-known ones.Finally, this paper is concluded in Section V.

II. RELATED WORK

To improve the performance of the Ncut criterion for theregion-based graphs, a solution was proposed in [22]. In thatmethod, each node of the region-based graph G is replacedby a complete graph Kc with c nodes. In the new graphGc, there exists an edge between all nodes of two Kc graphs


Fig. 3. (a) Two nodes of the original graph G , (b) corresponding nodes andedges in G2.

corresponding to two adjacent nodes of G, with weight equalto that of the associated edge in G. Also, the weights of alledges in Kc graphs are set equal to 1. With c = 2, two nodesof the original graph G and the corresponding nodes and edgesin G2 are illustrated in Fig. 3.(a) and Fig. 3.(b), respectively.

It is not hard to see that the weight and degree matrices ofGc are Wc = W ⊗�c×c and Dc = c(D⊗ Ic), where ⊗ denotesthe Kronecker product operator, �c×c is an all-one c×c matrixand Ic denotes the identity c × c matrix.

Intuitively, it seems that using Gc can improve the quality ofsegmentation, as each region corresponds to c nodes (insteadof a single node in G) and hence, isolated regions are morelikely to be separated. However, by proving the followingtheorem, we show that when the recursive two-way strategyis adopted, the problem of using Ncut for the region-basedgraphs cannot be resolved by employing Gc.

Theorem 1: For a graph G with n nodes and the corre-sponding Gc with c × n nodes we have

(a) If λ1 ≤ λ2 ≤ · · · ≤ λn are the eigenvalues of the general-ized eigenvalue problem (3), then the eigenvalues of

(Dc − Wc)x = λDc x (5)

are λ1, λ2, . . . , λn and λn+1 = λn+2 = · · · = λc×n = 1.(b) If W is not a constant matrix, i.e., W �= α�n×n for any

arbitrary positive real number α, and if wii ≥ wi j for1 ≤ i, j ≤ n, then we have λ2 < 1.

(c) If fi and gi denote the eigenvectors corresponding to aneigenvalue λi �= 1 in equations (3) and (5), respectively,then we have gi = fi ⊗ �c×1.

Proof : The proof of this theorem is provided inAppendix A. �

A constant weight matrix means that each node is assimilar to other nodes as to itself. In the context of imagesegmentation, this means that the whole image can be regardedas one segment. Thus, we exclude this trivial case. Nowrecall that the entry wi j of the weight matrix W = [wi j ]n×n

represents the similarity between the two regions i and j .Therefore, it must be wii ≥ wi j for all 1 ≤ i, j ≤ n [asfor (4)]. Since W satisfies the conditions of Theorem 1-(b),we have λ2 < 1 and hence Theorem 1-(a) implies thatthe second smallest eigenvalue of (5) is also λ2. Thus fromTheorem 1-(c), we conclude that the eigenvectors correspond-ing to the second smallest eigenvalues of the generalizedeigenvalue problems (3) and (5), i.e., f2 and g2, are relatedby g2 = f2 ⊗ �c×1. Clearly, regardless of the discretizationalgorithm, the output of segmentation process is the same withf2 and g2.

III. OUR APPROACH: INCLUDING THE SIZE OF REGIONS

IN THE COST FUNCTION

Let us have a closer look at the Ncut criterion to clarifyhow it results in clusters with high inter-cluster and lowintra-cluster similarities. Note that for any partitioning (A, A),assoc(A, V ) + assoc(A, V ) is equal to the constant valueW = ∑

i∈V∑

j∈V wi j and hence, we can rewrite (1) as

Ncut(A, A) = W × cut(A, A)

assoc(A, V ) × assoc(A, V ). (6)

Therefore, when minimizing Ncut(A, A), we actually try todecrease cut(A, A) and simultaneously increase the prod-uct of assoc(A, V ) and assoc(A, V ). Moreover, increasingassoc(A, V )×assoc(A, V ) means increasing both assoc(A, V )and assoc(A, V ) at the same time. In this way, we can interpret(6) as both assoc(A, V ) and assoc(A, V ) represent the inter-cluster similarity, and cut(A, A) stands for the intra-clustersimilarity of the regions A and A. Thus, the similarity withineach region is considered by the weights of the edges thathave at least one endpoint in that region. Similarly, in the“AverageCut” criterion [30] defined by

AverageCut(A, A) = cut(A, A)

|A| + cut(A, A)

|A| , (7)

the number of nodes in the partitions A and A (denoted by1

|A| and |A|, respectively) represent their corresponding inter-cluster similarity.

If the underlying graph is pixel-based, the nodes of the graphare of the same significance and consequently, the cost func-tions (6) and (7) lead to acceptable segmentation. In contrast,since the nodes of a region-based graph represent differentnumbers of pixels, and hence have different significances, thementioned criteria may lead to undesirable result (See Fig. 2).Although many cost functions can be proposed to consider thesize of regions in graph partitioning, it is essential to chooseone whose corresponding optimal or suboptimal solution canbe obtained by an efficient algorithm in a reasonable time.

An effective approach for partitioning a graph based on acost function like Ncut and AverageCut is one based on thespectrum of the adjacency matrix of the graph. Implementationof this spectral clustering scheme is easy considering the aimof standard linear algebraic methods [31]. Therefore, our goalis to define an appropriate cost function whose optimal orsuboptimal solution can be obtained by spectral clustering.In [31], the real-valued solution of Ncut (initially presentedin [8]) is rederived through a simpler way. Here, we extendthis approach to prove Theorem 2 which characterizes a classof cost functions that can be optimized by using spectralclustering approach.

Theorem 2: For a given graph G with node set V ={1, 2, . . . , n} and weight matrix W , let S = [si j ] denote ann ×n diagonal matrix with positive main diagonal entries. Wedefine

s(B) =∑

i∈B

sii

1Note that A and A are two subsets of the graph nodes and |B| denotes thecardinality of the set B .


for each B ⊂ V . The second smallest eigenvector of thegeneralized eigenvalue problem

(D − W )x = λSx (8)

is the relaxed indicator vector corresponding to

arg minA⊂V

cut(A, A)

s(A) × s(A). (9)

�Proof: For a given A ⊂ V , let f = ( f1, . . . , fn)T ∈ R

n bea binary vector with entries

fi =

⎧⎪⎪⎨

⎪⎪⎩

√s(A)s(A) if i ∈ A

−√

s(A)

s(A)if i ∈ A

. (10)

Also, L denotes the laplacian matrix of G, i.e., L � D − W .Following the approach of [31], we can write

f T L f = 1

2

n∑

i, j=1

wi j ( fi − f j )2

= 1

2

∑

i∈A, j∈A

wi j

⎛

⎝√

s(A)

s(A)+

√s(A)

s(A)

⎞

⎠2

+1

2

∑

i∈A, j∈A

wi j

⎛

⎝−√

s(A)

s(A)−

√s(A)

s(A)

⎞

⎠2

= cut(A, A)

⎛

⎝√

s(A)

s(A)+

√s(A)

s(A)

⎞

⎠2

= tr2(S)cut (A, A)

s(A) × s(A),

where tr(S) is the trace of the matrix S. Moreover, we have

(S f )T� =

n∑

i=1

sii fi =∑

i∈A

sii

√s(A)

s(A)−

∑

i∈A

sii

√s(A)

s(A)= 0,

and also,

f T S f =n∑

i=1

sii f 2i =

∑

i∈A

siis(A)

s(A)+

∑

i∈A

siis(A)

s(A)= tr(S) .

Therefore, solving the optimization problem (9) is equivalentto finding

arg minf as in (10) for some A ⊂ V

S f ⊥� and f TS f = tr(S)

f T L f .

Note that finding the optimal solution for the above optimiza-tion problem is very hard as it is a discrete problem, i.e., theelements of a vector f are allowed to take only two distinctvalues. By relaxing fi to take any real value, the problembecomes one of finding

arg minf ∈Rn : S f ⊥� and f T

S f = tr(S)

f T L f . (11)

Substituting g = S12 f , we obtain the relaxed problem

arg ming∈Rn : g⊥S

12 � and ‖g‖2= tr(S)

gT S−1/2 LS−1/2 g, (12)

where S−1/2LS−1/2 is a real symmetric matrix.Lemma 1: The matrix S−1/2 LS−1/2 is positive semi-

definite.Lemma 2: The smallest eigenvalue of S−1/2LS−1/2 is zero

and its corresponding eigenvector is S1/2�.

The proofs of the above Lemmas are given in Appendix B.Now, recall the following property of Rayleigh quotient [32].

Lemma 3: Let A be a real symmetric matrix with theeigenvalues λ1 ≤ λ2 ≤ · · · ≤ λn and the correspondingeigenvectors x1, x2, . . . , xn . If x is orthogonal to the j − 1smallest eigenvectors, i.e., x1, x2, . . . , x j−1, then

arg minx

xT AxxT x

= x j

and

minx

xT AxxT x

= λ j .

According to Lemmas 1 and 2, the optimization problem(12) is in the form of Lemma 3 and hence, its solution is thesecond smallest eigenvector of the eigenvalue problem

S−1/2 LS−1/2 g = λg.

Replacing f by S−1/2 g, we can write S−1/2 L f = λS1/2 fand

L f = λS f . (13)

Therefore, the optimum solution of the problem (11) is thesecond smallest eigenvector of (13). �

Note that minimizing

cut(A, A)

s(A) × s(A)(14)

means decreasing cut(A, A) (low intra-cluster similarity) andincreasing s(A)× s(A). On the other hand, since s(A)+ s(A)is a constant value independent of A, increasing s(A) × s(A)implies increasing both s(A) and s(A). In this way, if s(A)and s(A) indicate the inter-cluster similarity of A and A,we can satisfy both fundamental principles of clustering byminimizing (14). In doing so, the entry sii should representthe contribution of the node i in the inter-cluster similarityof the cluster (A or A) which contains node i . Since the roleof the matrix S in the cost function (14) is evaluating theinter-cluster similarity, one can define an appropriate matrix Sfor a specific task and simply derive a suboptimal solution bylinear algebraic methods. For instance, if we set S = In , thenthe inter-cluster similarity of each cluster will be equal to thenumber of its nodes, i.e., s(A) = |A| and hence, we have

cut (A, A)

s(A) × s(A)= |V | × AverageCut(A, A).

Thus, S = In leads to the AverageCut criterion. In the sameway, Ncut will be derived if we set S = D. Therefore, theclass of cost functions defined by different S matrices containsboth the Ncut and AverageCut criteria. Considering the main


concern of this paper, we should define sii so that the sizeof the region corresponding to node i is somehow taken intoaccount. To do so, we adopt the simplest way by definingsii = Ni , where Ni stands for the number of pixels in theregion i .

With this definition of S, the feature space information ofpixels is utilized for representing intra-cluster similarity inthe numerator of (9) and the size of the regions is used forreflecting the inter-cluster similarity in the denominator. Inother words, the right and left hand sides of the generalizedeigenvalue problem (8) are influenced by the inter-cluster andintra-cluster similarity, respectively. By using this new costfunction, the region 5 of Fig. 2.(a) is separated in the firststep of partitioning.

IV. QUANTITATIVE EVALUATION OF THE

PROPOSED COST FUNCTION

In this section, we present experimental results and comparethe performance of the proposed cost function, i.e.,

cut(A, A)

(∑

i∈A Ni ) × (∑

i∈A Ni ),

with that of the Ncut and AverageCut. In the implementationof Mean-shift algorithm, EDISON system (Edge Detection andImage SegmentatiON) [26], which applies the information ofL∗u∗v∗ color space, was employed. The bandwidth parameterswere set h = (hs, hr ) = (7, 6.5) and the minimum region sizewas set at most 200 pixels, depending on the image details.

In order to compare the results of different cost functions,a quantitative measure is required for evaluating the quality ofthe resulting segmentations. Most of the evaluation measuresare based on a comparison between the resulting segmentationwith a reference one, called ground truth. Here, we employedthe normalized partition distance [33], [34]. For two givensegmentations of an image, partition distance shown by dsym ,is defined as the minimum number of image pixels that mustbe discarded so that the two induced segmentations becomeidentical. The measure normalized partition distance is equalto dsym/(N − 1), where N denotes the number of pixels inthe image.

The test images were chosen from the Berkeley segmenta-tion database [35], which contains hand-labeled segmentationsof the images from 30 human subjects. Most of the imagesin this database include complex textures, which are suitableto compare texture-based image segmentation methods. Butsince we focused on the performance of some cost functionsbased only on the color information, 25 low texture imageswere selected from this database. Moreover, since the imagesare to be segmented by partitioning their region-based graphs,the resulting boundaries are limited to those initially obtainedby the Mean-shift algorithm. Hence, we constructed our ownground truth for the images based on the output regions ofthe Mean-shift algorithm. Note that it is desirable to separatesignificant segments of the image in the first recursions of thealgorithm. Hence, only the most salient regions of the imageswere considered in the hand-labeling procedure. The secondand the third columns of Fig. 6 illustrate the result of the

Fig. 4. The mean value of the normalized partition distance for 25 selectedimages with the proposed cost function, Ncut, and AverageCut as a functionof dI .

Fig. 5. Normalized partition distance for 25 selected images with theproposed cost function and Ncut.

Mean-shift segmentation and the corresponding hand-labeledground truth of some of the selected images.

The weight wi j of the region-based graph, which illustratesthe likelihood that the regions i and j lie in a cluster, is usuallyconsidered as a decreasing function of their distance in thefeature space. We used the weight function given in (4) whereFi = [Li , ui , vi ] stands for the mean color vector of region iin the L∗u∗v∗ color space. To achieve a fair comparisonbetween cost functions under evaluation, the simulation wasaccomplished for different values of dI .

For each value of dI , the proposed cost function, Ncutand AverageCut were applied to region-based graph anddsym/(N − 1) was computed for the resulting segmentationsand the ground truth. The mean of dsym/(N − 1) over 25selected images is illustrated in Fig. 4 as a function of dI . Wecan observe that the proposed cost function shows a betterperformance for all values of dI which have been considered.Also Fig. 4 reveals that when using the proposed cost function,300 is an appropriate choice for the parameter dI . One can findthe result of segmentation for some of the images in Fig. 6.Also, the normalized partition distances with the proposedcost function and Ncut are compared in Fig. 5 for all 25employed images. For this comparison, the best value of dI

for each cost function (i.e., dI = 200 for the Ncut and


Fig. 6. The results of color image segmentation based on the Ncut and proposed cost function. First column: the original test images. Second column: theoutput of the Mean-shift algorithm. Third column: desirable segmentation (constructed ground truth). Fourth column: segmentation by the Ncut criterion withdI = 200. Fifth column: segmentation by the proposed cost function with dI = 300.

dI = 300 for the proposed cost function) is considered. Theresults of segmentation for all 25 images are provided in thesupplementary figures.

Clearly, the proposed cost function cannot outperform theNcut and provide the ideal segmentation in all cases. Supposethat A and A are two salient segments, i.e., S(A) and S(A)


Fig. 7. (a) Original image, (b) output of Mean-shift algorithm (268 regions),(c) segmentation using the proposed cost function (after 9 recursions).

are large. Clearly, the product S(A) × S(A) in the denom-inator of the proposed cost function increases the chance ofbipartitioning the graph into (A, A). Sometimes the Mean-shiftalgorithm may fragment a large segment of the image, say A,to many small regions [See the forest in Fig. 7.(a)]. In suchcases, since there are many edges between A and A, the valueof cut (A, A) might be large so that the product S(A) × S(A)

cannot make cut (A,A)

S(A)×S(A)small enough and hence, A and A

do not minimize the cost function. In Fig. 7.(c) we expectthe algorithm to separate the whole forest region as a salientsegment in the initial recursions of the segmentation process,but the output is not so.

V. CONCLUSION

The main advantage of the Ncut criterion in image segmen-tation using pixel-based graphs, i.e., preventing separation ofisolated small components, may become a disadvantage whenit is applied to region-based graphs. This is because in region-based graphs, a single node may represent a large region of theimage which must be considered as a major segment. In orderto obviate this shortcoming, we considered the size of regionsin the cost function. Some cost functions were characterizedfor which the suboptimal solution could be found by solving ageneralized eigenvalue problem (the same as Ncut). Then, oneof these cost functions was devised so that the size of regionsinfluence the inter-cluster similarity of the regions. Simulationshowed that the proposed cost function outperforms Ncut andAverageCut.

APPENDIX A

Before proving Theorem 1, we mention some propertiesof the Kronecker product. Throughout this appendix, |0 and 0denote all-zero matrix and vector/scalar, respectively.

Lemma A-1: Given the matrices Am×p , Br×s , Cp×q andDs×t and the scaler α, some basic properties of the Kroneckerproduct are [36]:(a) If A �= |0 and B �= |0, then A ⊗ B �= B ⊗ A.(b) α(A ⊗ B) = αA ⊗ B = A ⊗ αB .(c) A ⊗ B = |0 iff A = |0 or B = |0.(d) If R is a matrix with the same dimensions as A, then

(A + R) ⊗ C = A ⊗ C + R ⊗ C .(e) (A ⊗ B)(C ⊗ D) = AC ⊗ B D.(f) For p × p matrices A1, A2, . . . , Ak and q × q matrices

B1, B2, . . . , Bk , we have det(A1 ⊗ B1 + A2 ⊗ B2 + · · · +Ak ⊗ Bk) = det(B1 ⊗ A1 + B2 ⊗ A2 + · · · + Bk ⊗ Ak). �

Proof of Theorem 1-(a): Define h(λ) � det(Dc − Wc − λDc)and Gλ � (1 − λ)D. The eigenvalues of the problem

(Dc − Wc)x = λDc x are roots of the equation h(λ) = 0.From Wc = W ⊗�c×c , Dc = c (D ⊗ Ic) and Lemma A-1, wecan write

h(λ) = det(cGλ ⊗ Ic − W ⊗ �c×c

)

= det(cIc ⊗ Gλ − �c×c ⊗ W

)

= det

⎛

⎜⎜⎜⎜⎝

⎡

⎢⎢⎢⎢⎣

cGλ − W −W . . . −W

−W...

... −W−W . . . −W cGλ − W

⎤

⎥⎥⎥⎥⎦

⎞

⎟⎟⎟⎟⎠

= det

⎛

⎜⎜⎜⎜⎜⎜⎝

⎡

⎢⎢⎢⎢⎢⎢⎣

cGλ |0n×n . . . |0n×n −W

|0n×n...

...... |0n×n

...|0n×n . . . |0n×n cGλ −W−cGλ . . . −cGλ −cGλ cGλ − W

⎤

⎥⎥⎥⎥⎥⎥⎦

⎞

⎟⎟⎟⎟⎟⎟⎠

= det

⎛⎜⎜⎜⎜⎜⎜⎝

⎡⎢⎢⎢⎢⎢⎢⎣

cGλ |0n×n . . . |0n×n −W

|0n×n...

...... |0n×n

...|0n×n . . . |0n×n cGλ −W|0n×n . . . |0n×n |0n×n c(Gλ − W )

⎤⎥⎥⎥⎥⎥⎥⎦

⎞⎟⎟⎟⎟⎟⎟⎠

.

(A-1)

Now, we define S � c(Gλ − W ) and

P �

⎡

⎢⎢⎢⎢⎣

cGλ |0n×n . . . |0n×n

|0n×n...

... |0n×n|0n×n . . . |0n×n cGλ

⎤

⎥⎥⎥⎥⎦, Q �

⎡

⎢⎣−W

...−W

⎤

⎥⎦ ,

which are respectively n × n, (c − 1)n × (c − 1)n and(c − 1)n × n matrices. Thus (A-1) gives

h(λ) = det

([P Q

|0n×(c−1)n S

])

= det(P). det(S)

=[

cn(1 − λ)nn∏

i=1

di

]c−1

× [cn. det(Gλ − W )

](A-2)

where di is the i -th main diagonal entry of D. Noting (A-2)we see that the roots of h(λ) = 0 consists of the roots of theequation det

((1 −λ)D − W

) = 0 [i.e., the eigenvalues of (3)]and λ = 1 with multiplicity (c − 1)n. �

Proof of Theorem 1-(b): Multiplying both sides of (3) byD−1 from the left, we can conclude that the eigenvalues of(3), i.e., λ1, λ2, . . . , λn , are the same as the eigenvalues of thematrix D−1 L. On the other hand, it is easy to verify that theeigenvalues of D−1 L and D−1/2 L D−1/2 are the same [31].D−1/2 L D−1/2 is an n × n symmetric matrix with entries:

(D−1/2 L D−1/2)i j =

⎧⎪⎨

⎪⎩

1 − wiidi

, if i = j ,

− wi j√di d j

, if i �= j ,. (A-3)


It is well-known that the sum of the eigenvalues of a matrixA is equal to its trace tr(A) [32]. Thus, from (A-3) we have

n −n∑

i=1

wii

di=

n∑

i=1

λi . (A-4)

Since wii ≥ wi j for 1 ≤ j ≤ n implies that di = ∑nj=1 wi j ≤

nwii , we have wiidi

≥ 1n . Moreover, W �= α�n×n means that

there exists an entry wi0 j0 which is strictly smaller than wi0 i0and hence,

wi0 i0di0

> 1n . Therefore, we have

n∑

i=1

wii

di>

n∑

i=1

1

n= 1. (A-5)

Now we can show that λ2 ≥ 1 cannot be true. This is becauseif we assume that λ2 ≥ 1, then the fact that λ1 = 0 [31] andλ2 ≤ · · · ≤ λn and (A-4) imply that

∑ni=1

wiidi

≤ 1, whichcontradicts (A-5). �Proof of Theorem 1-(c): Since fi is the eigenvector corre-sponding to eigenvalue λi of (3), we have (D − λi D − W )fi = 0. Thus we can use Lemma A-1 to write

0 = c(D − λi D − W ) fi ⊗ �c×1

= c(1 − λi )D fi ⊗ Ic�c×1 − cW fi ⊗ �c×1

= c(1 − λi )D fi ⊗ Ic�c×1 − W fi ⊗ �c×c�c×1

= [c(1−λi)D ⊗ Ic] ( fi ⊗ �c×1)−(W ⊗ �c×c)( fi ⊗ �c×1)

= [c(1 − λi )D ⊗ Ic − W ⊗ �c×c

]( fi ⊗ �c×1)

= (Dc − λi Dc − Wc)( fi ⊗ �c×1). (A-6)

Clearly, from (A-6) we can conclude that fi ⊗ �c×1 is theeigenvector corresponding to eigenvalue λi of (5). �

APPENDIX B

Proof of Lemma 1: Since S is a diagonal matrix withpositive main diagonal entries, it is positive semi-definite. Thisis also the case for S−1/2. On the other hand, L is the laplacianmatrix of the graph which is positive semi-definite [31]. It isknown that if P and Q are positive semi-definite matrices,P Q P and Q P Q are also positive semi-definite [32]. �

Proof of Lemma 2: If S−1/2 LS−1/2x = λx, then it must beLS−1/2x = λS1/2x. Substituting x by S1/2

� gives L� = λS�.Since L� is a zero vector, S1/2

� is the eigenvector of thematrix S−1/2 LS−1/2 corresponding to the eigenvalue λ = 0.Furthermore, S−1/2 LS−1/2 is a positive semi-definite matrix(Lemma 1) and its eigenvalues are non-negative. Therefore,λ = 0 is the smallest eigenvalue of S−1/2 LS−1/2. �

REFERENCES

[1] N. Pal and S. Pal, “A review on image segmentation techniques,” PatternRecognit., vol. 26, no. 9, pp. 1277–1294, 1993.

[2] L. Lucchese and S. K. Mitra, “Advances in color image segmentation,”in Proc. Global Telecommun. Conf., 1999, pp. 2038–2044.

[3] L. Lucchese and S. K. Mitra, “Color image segmentation: A state-of-the-art survey,” in Proc. Indian Nat. Sci. Acad., vol. 67. 2001, pp. 207–221.

[4] S. Bhattacharya, “A brief survey of color image preprocessing andsegmentation techniques,” J. Pattern Recognit. Res., vol. 1, no. 1,pp. 120–129, 2011.

[5] J. McQueen, “Some methods for classification and analysis of multivari-ate observations,” in Proc. 5th Berkeley Symp. Math. Statist. Probab.,vol. 1. 1967, pp. 281–296.

[6] Y. Cheng, “Mean shift, mode seeking, and clustering,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 17, no. 8, pp. 790–799, Aug. 1995.

[7] L. Shafarenko, M. Petrou, and J. Kittler, “Automatic watershed segmen-tation of randomly textured color images,” IEEE Trans. Image Process.,vol. 6, no. 11, pp. 1530–1544, Nov. 1997.

[8] J. Shi and J. Malik, “Normalized cuts and image segmentation,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 8, pp. 888–905,Aug. 2000.

[9] S. Makrogiannis, G. Economou, and S. Fotopoulos, “A region dissim-ilarity relation that combines feature-space and spatial information forcolor image segmentation,” IEEE Trans. Syst., Man, Cybern. B, Cybern.,vol. 35, no. 1, pp. 44–53, Feb. 2005.

[10] R. Bajcsy, S. W. Lee, and A. Leonardis, “Detection of diffuse andspecular interface reflections and inter-reflections by color image seg-mentation,” Int. J. Comput. Vis., vol. 17, no. 3, pp. 241–272, Mar. 1996.

[11] B. A. Maxwell and S. A. Shafer, “Physics-based segmentation of com-plex objects using multiple hypotheses of image formation,” Comput.Vis. Image Understand., vol. 65, no. 2, pp. 269–295, Feb. 1997.

[12] S. X. Yu and J. B. Shi, “Segmentation given partial grouping con-straints,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 2,pp. 173–183, Feb. 2004.

[13] Z. Wu and R. Leahy, “An optimal graph theoretic approach to dataclustering: Theory and its application to image segmentation,” IEEETrans. Pattern Anal. Mach. Intell., vol. 15, no. 11, pp. 1101–1113,Nov. 1993.

[14] O. Veksler, “Image segmentation by nested cuts,” in Proc. IEEE Int.Conf. CVPR, Jun. 2000, pp. 339–344.

[15] S. Wang and J. M. Siskind, “Image segmentation with ratio cut,” IEEETrans. Pattern Anal. Mach. Intell., vol. 25, no. 6, pp. 675–690, Jun. 2003.

[16] I. J. Cox, S. B. Rao, and Y. Zhong, “Ratio regions: A technique forimage segmentation,” in Proc. ICPR, Aug. 1996, pp. 557–564.

[17] J. Carreira and C. Sminchisescu, “CPMC: Automatic object segmenta-tion using constrained parametric min-cuts,” IEEE Trans. Pattern Anal.Mach. Intell., vol. 34, no. 7, pp. 1312–1328, Jul. 2012.

[18] Y. Huang, Q. Liu, F. Lv, Y. Gong, and D. N. Metaxas, “Unsupervisedimage categorization by hypergraph partition,” IEEE Trans. PatternAnal. Mach. Intell., vol. 33, no. 6, pp. 1266–1273, Jun. 2011.

[19] H. Lu, Y.-P. Tan, and X. Xue, “Real-time, adaptive, and locality-basedgraph partitioning method for video scene clustering,” IEEE Trans.Circuits Syst. Video Technol., vol. 21, no. 11, pp. 1747–1759, Nov. 2011.

[20] A. Levinshtein, C. Sminchisescu, and S. Dickinson, “Optimal image andvideo closure by superpixel grouping,” Int. J. Comput. Vis., vol. 100,no. 1, pp. 99–119, Oct. 2012.

[21] D. S. Hochbaum, “Polynomial time algorithms for ratio regions anda variant of normalized cut,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 32, no. 5, pp. 889–898, May 2010.

[22] W. Tao, H. Jin, and Y. Zhang, “Color image segmentation based onmean shift and normalized cuts,” IEEE Trans. Syst., Man, Cybern. B,Cybern., vol. 37, no. 5, pp. 1382–1389, Oct. 2007.

[23] C. Fowlkes, S. Belongie, F. R. K. Chung, and J. Malik, “Spectralgrouping using the nystrom method,” IEEE Trans. Pattern Anal. Mach.Intell., vol. 26, no. 2, pp. 214–225, Feb. 2004.

[24] E. Sharon, A. Brandt, and R. Basri, “Fast multiscale image segmenta-tion,” in Proc. IEEE Conf. CVPR, vol. 1. Jun. 2000, pp. 70–77.

[25] W.-Y. Chen, Y. Song, H. Bai, C.-J. Lin, and E. Y. Chang, “Parallelspectral clustering in distributed systems,” IEEE Trans. Pattern Anal.Mach. Intell., vol. 33, no. 3, pp. 568–586, Mar. 2011.

[26] D. Comaniciu and P. Meer, “Mean shift: A robust approach towardfeature space analysis,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24,no. 5, pp. 603–619, May 2002.

[27] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk,“SLIC superpixels compared to state-of-the-art superpixel methods,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 6, no. 1, pp. 2274–2282,Dec. 2011.

[28] L.-X. Liu, G.-Z. Tan, and M. S. Soliman, “Color image segmentationusing mean shift and improved ant clustering,” J. Central South Univ.Technol., vol. 19, no. 4, pp. 1040–1048, Apr. 2012.

[29] A. Levinshtein, A. Stere, K. N. Kutulakos, D. J. Fleet, S. J. Dickinson,and K. Siddiqi, “TurboPixels: Fast superpixels using geometric flows,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 12, pp. 2290–2297,Dec. 2009.

[30] S. Sarkar and P. Soundararajan, “Supervised learning of large percep-tual organization: Graph spectral partitioning and learning automata,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 5, pp. 504–525,May 2000.

[31] U. von Luxburg, “A tutorial on spectral clustering,” Statist. Comput.,vol. 17, no. 4, pp. 395–416, 2007.


[32] H. Lütkepohl, Handbook of Matrices. Chichester, U.K.: Wiley, 1996.[33] A. Almudevar and C. Field, “Estimation of single generation sibling

relationships based on DNA markers,” J. Agricult., Biol., Environm.Statist., vol. 4, pp. 136–165, 1999.

[34] J. S. Cardoso and L. Corte-Real, “Toward a generic evaluation ofimage segmentation,” IEEE Trans. Image Process., vol. 14, no. 11,pp. 1773–1782, Nov. 2005.

[35] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of humansegmented natural images and its application to evaluating segmentationalgorithms and measuring ecological statistics,” in Proc. 8th ICVV,vol. 2. Jul. 2001, pp. 416–423.

[36] R. A. Horn and C. R. Johnson, Topics in Matrix Analysis. Cambridge,U.K.: Cambridge Univ. Press, 1991.

Alireza Rezvanifar received the B.Sc. and M.Sc.degrees in electrical engineering from the NajafabadBranch of Islamic Azad University and IsfahanUniversity of Technology, Iran, in 2006 and 2010,respectively. Since 2010, he has been a Researcherwith the Information and Communication Tech-nology Institute, Isfahan University of Technology.His research interests include signal and imageprocessing.

Mohammadali Khosravifard (M’07) received theB.Sc., M.Sc., and Ph.D. degrees in electrical engi-neering from Shiraz University, Sharif University ofTechnology, and Isfahan University of Technology,Iran, in 1996, 1998, and 2004, respectively. Since2004, he has been with the Department of Electricaland Computer Engineering, Isfahan University ofTechnology, where he is currently an AssociateProfessor. His research interests include informationtheory and image processing.

including the size of regions in image segmentation by region-based graph

Documents