dynamic measurement of computer generated image segmentations

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-7, NO. 2, MARCH 1985

Dynamic Measurement of Computer GeneratedImage Segmentations

MARTIN D. LEVINE, SENIOR MEMBER, IEEE, AND AHMED M. NAZIF

Abstract-This paper introduces a general purpose performance mea-surement scheme for image segmentation algorithms. Performanceparameters that function in real-time distinguish this method from pre-vious approaches that depended on an a priori knowledge of the correctsegmentation. A low level, context independent definition of segmenta-tion is used to obtain a set of optimization criteria for evaluating per-formance. Uniformity within each region and contrast between adjacentregions serve as parameters for region analysis. Contrast across linesand connectivity between them represent measures for line analysis.Texture is depicted by the introduction of focus of attention areas asgroups of regions and lines. The performance parameters are then mea-sured separately for each area. The usefulness of this approach lies inthe ability to adjust the strategy of a system according to the varyingcharacteristics of different areas. This feedback path provides themeans for more efficient and error-free processing. Results from areaswith dissimilar properties show a diversity in the measurements that isutilized for dynamic strategy setting.

Index Terms-Edge detection, expert system, image contrast, imagesegmentation, image uniformity, partial segmentation, region analysis.

I. INTRODUCTIONMsEASURING performance is a key factor in providing a

feedback path by which a system can modify its strategyduring processing. For systems with well defined goals, per-formance parameters can be found that measure the distancefrom the goal at any point in time. These parameters can thenbe used to optimize the path towards that goal. Unfortunately,image segmentation systems do not fall into this category, atleast not at the low level of processing. A goal can only bedefined precisely in high level terms. Segmentation of theimage into regions that correspond to objects in the scene, orthe isolation of a particular object from the background areexamples of these.In terms of low level image segmentation, we can only hope

for a fuzzy definition of the goal. The primary reference usedto evaluate the final result is that of human performance insegmenting the image. We can define a segmentation as a parti-tion of the image into regions that are uniform among them-selves, and that bear contrast to their adjacent neighbors. Uni-formity and contrast are measured in terms of a set of low levelfeatures that can be evaluated over the image. For example,these features can be the average grey level intensities of the

Manuscript received January 18, 1984; revised September 25, 1984.Recommended for acceptance by Ruzena Bajcsy. This research wassupported in part by the Natural Sciences and Engineering ResearchCouncil under Grant A4156, and in part by the Department of Educa-tion, FCAC, P.Q., Canada, under Grant EQ-633.M. D. Levine is with the Computer Vision and Robotics Laboratory,

Department of Electrical Engineering, McGill University, Montreal, P.Q.,Canada.A. M. Nazif is with the Department of Electrical Engineering, Univer-

sity of Cairo, Cairo, Egypt.

regions in black and white or color spaces. Two factors arenot accounted for by this definition, although they do affecthuman evaluation: the presence of texture in an image and theexistence of lines as independent separate entities. A third im-portant aspect is context which is assumed to function at ahigher level.The problem, then, is to find a set of measures that can be

computed for a segmentation of an image, and that closelysimulate the judgment rendered by a human evaluating thesame segmentation. In earlier work, we presented a measureof segmentation that compared two image partitions [9] . Oneof these is produced by a human, and corresponds to the ob-jects or surfaces in the scene. The other represents the outputof the segmentation algorithm to be tested. A two-dimensionaldistance measure quantifies the difference between the two,and hence provides an estimate of the error in a computedpartition, relative to the man-made one. Therefore, it can beused to evaluate the performance of a particular algorithm, orto compare different outputs. More importantly, the measurecould be employed in learning, or in the case of a rule basedsystem, in building the model by testing the rules.Other measures that are based on the idea of comparing two

partitions have been described in the literature. The methodintroduced by Coleman -computes a statistical measure fromthe joint histogram of the segmented images [4]. In effect,this comparison measure estimates the number of misclassifiedimage points in one output relative to the other. The disad-vantage of this method lies in the loss of spatial information incomputing the histogram. This leads to cases where the mea-sure does not seem to agree well with human observation. Tocorrect for such errors, a more complex measure has been de-veloped that compares the manual partition of cell images withcomputed ones [21] . The position of misclassified pixels istaken into account, as well as their number. An estimate ofthe error is taken to be proportional to the sum of the distancesbetween pixels that have been incorrectly assigned to a par-ticular class and the nearest pixels that actually belong to thatclass. The method appears to perform well for the cell imageswhich have well defined classifications for the pixels (e.g.,nucleus, cytoplasm, red cells, and background). In the absenceof this contextual information, the measure is not computablebecause the classes are not defined. In contrast to the above,the procedure described in [9] provides a general purpose lowlevel comparison measure that can be applied to any class ofimages.A different type of performance measurement has been de-

veloped in conjunction with edge based segmentation tech-

0162-8828/85/0300-0155$01.00 © 1985 IEEE

155


Fig. 1. Two measures of segmentation.

niques. Fram and Deutsch [5] computed two measures thatevaluate the output of edge detectors. The first estimated thefraction of correctly detected edge points in a synthesizedimage of an edge in the presence of noise. The second com-

puted the fraction of the actual edge that was covered by de-tected pixels. Their results show the deterioration in the per-

formance of edge detectors with noise. They were also able torank several mask operators and demonstrate the consistentsuperiority of some schemes over others.Abdou and Pratt [1] employ an analytical approach to study

the sensitivity of detectors to edge orientation. In addition,they used a figure of merit for edge quality [1 8] that is basedon the displacement of the detected edge point from the idealedge position. As with the previous method, this requries theknowledge of the correct location of the edge in the synthesizedtest images. The results show an ordering of the edge detec-tors that is quite similar to that obtained by Fram and Deutsch

[5].Bryant and Bouldin [2] proposed a measure that is based

on the correlation of the output of edge detectors with an

"ideal" output, presumably produced by hand. In an effort toeliminate the need for the knowledge of the correct output,they described a second measure that computes a consensus

of several edge detectors on the presence or absence of edgepoints. Each detector is then evaluated on the basis of howwell it conforms to (or differs from) that consensus. The re-

sults are sensitive to the subset of operators used to computethe consensus. If a consistent error is part of the consensus,

the error will not be detected.Local edge coherence was used as the basis for the evaluation

given by Kitchen and Rosenfeld [6]. The method evaluatesthe consistency of the gradient directions at pixels in thevicinity of each edge point. Detected edges are rewarded forcontinuity and thinness. The method suffers from a presump-

tion that the detected edges are in their proper position. There-fore, it does not penalize consistent errors in mislocating edgepoints. The authors note that this renders the method validonly as a supplement to existing measures. Test results on

computer generated images of edges with superimposed noisegenerally agree with previous findings with regard to rating var-

ious edge mask operators.The approaches mentioned so far, with the exception of the

last two edge evaluation approaches, describe comparison mea-

sures that require a reference segmentation in order to evaluateperformance. However, in automatic processing, the humanelement is usually not available, rendering these measures use-less for real-time utilization. Consensus measurement and edgecoherence methods realize the importance of eliminating sucha dependency. Nevertheless, they do not provide comprehen-sive measures that can function without the knowledge of thecorrect segmentation, or some other precomputed reference(as in the consensus case). What is required is a comparison be-tween the provided measurements (the stored image) and thecomputed partition (the intermediate and final output). Fig.1 illustrates the difference between a real-time measure, andone that is used for testing and comparison.The definition of segmentation given above provides a clue to

the design of an appropriate real-time measure. What is neededare estimates of uniformity within regions, contrast acrossregions, as well as provision for lines and texture. In this paper,we introduce the optimization criteria that are involved inmeasuring the performance of image segmentation systems.These are intended as general purpose parameters that can beused (all or the appropriate subset) in evaluating, as well asguiding, the processing of any partitioning algorithm. Theyapply to region based, line based, and texture based approachesto segmentation.- In particular, the set of measures detailedhere was designed to provide the means for setting the process-ing strategy of the rule based image segmentation system pre-sented in [9], [17] . The latter is introduced briefly in the nextsection, with particular emphasis on the data structure and thestrategy requirements. The remaining sections discuss the per-formance measures.

II. RULE BASED IMAGE SEGMENTATION

In order to identify the contents of an image, a partitionmust first be produced that corresponds to the objects in thescene. This can be done by finding the regions that partiallyor totally correspond to the surfaces of these objects. Alterna-tively, the lines that define the boundaries of these surfacescould be detected. These two approaches have been extensivelyused for image segmentation, but never simultaneously. Re-gion analysis depends on optimizing the uniformity of the parti-tioned segments, as well as maximizing the contrast betweenthem. Line analysis is accomplished by detecting local discon-tinuities in the features. Since they are based on differentparadigms, both regions and lines should be employed as partof the data structure. Thus, the information from one sourcecan be used to benefit the processing of the other.

Initially, the image is divided (perhaps arbitrarily) into a setof regions, thereby forming a region map. Also, lines are tracedat points of high discontinuity in the measured features toform a separate line map. The analysis consists of repeatedlymerging and splitting existing regions, as well as adding, delet-ing, and joining lines. A third entry is included in the data andrepresents the focus of attention areas. These are composed ofgroups of regions and lines that correspond to one of threeinteresting phenomena in the image. The first consists of agroup (usually a small number) of large regions that are ad-jacent and highly uniform. They result in areas that are labeledas smooth. Next are the textured areas which are represented

156

LEVINE AND NAZIF: COMPUTER GENERATED IMAGE SEGMENTATIONS

LONG

TERM

MEMORY

=:>t SUPERVISOR <==j>

FOCUS OF WATTENTION

REGIONANALYZER

LINEANALYZER -

AREA vANALYZER

SHORT

TERM

MEMORY

Fig. 2. Block diagram of the rule based segmentation system.

by a large number of small adjacent regions that bear contrastto each other. Finally, bounded areas are formed when a longline closes back on itself forming a loop. These three types ofareas serve two purposes. One is to guide the processing to theinformation-rich parts of the image, and the other is to providevaluable information for the ensuing image interpretation sys-

tem. Processing starts with an initial configuration of areas

that covers the whole image. As the analysis progresses, areas

are added, deleted, or modified to conform with the changingproperties of regions and lines.Knowledge is applied in the form of production rules about

the low level features of regions, lines, and areas [12]. Theserules modify the data in a direction that better conforms withthe world model. The conditions in a rule may be satisfied bymore than one data entry. Creating a processing strategy cor-

responds to defining which rules to apply and which data entryto test at each point in time. The details of the system, in-cluding segmentation results, appear in the other papers [11],[12], [17]. Here, we only address the problem of segmenta-tion measures.

Fig. 2 shows a block diagram of the system in which threetypes of processing modules are distinguishable. The first in-cludes all the processing modules which employ expert knowl-edge. They apply the low level segmentation criteria embodiedin the rules to the image data stored in the short term memory(STM). Each process has its own rule set which constitutespart of the model stored in the long term memory (LTM). Inthis system, the REGION, LINE, and AREA processes will bereferred to as the KNOWLEDGE MODULES. The second type,the FOCUS OF ATTENTION process, assumes the task of select-ing the particular data entries against which the knowledgerules are to be matched. This is accomplished by using focusof attention rules, which are also stored in the LTM. The lasttype is the SUPERVISOR, which is the process responsible forestablishing the strategy to be executed by the other two typesof modules. It matches its own meta-rules to the data, andconsequently determines the order of activating the other pro-cessing modules. It can also modify the priority of knowledge

rules and focus of attention rules to reflect a particular strategy.The first two types are thus rule based, data-driven processes,whose actions alter the state of the data according to the criteriaspecified by the model. The SUPERVISOR is also rule-basedand data-driven. Its actions, however, alter the system strategyby specifying priorities among the processing modules, as wellas among the rules belonging to each.A rule cycle begins by matching the conditions of a par-

ticular rule to specific data entries. These conditions requirethat the features of a region, line, or area be tested for thepresence of a certain data configuration. Therefore, eachKNOWLEDGE MODULE must have selected for it the specificregion, line, and area to be considered at any point in the pro-cessing. Conceptually, the system is searching the image forthe occurrence of the data configuration represented by a rule.It is the order of that search that must be established here.This translates into the problem of selecting the next area ofattention to process, and specifying the order within this areain which the regions and lines are to be visited.Image processing systems in the past have not seriously ad-

dressed the problem of determining the order in which an imageis to be processed. However, there is ample proof that such anordering affects the results of the analysis. This is evident, forexample, in region growing systems in which initial regions aremerged in a prespecified order to form larger regions. It hasbeen shown [13] that changing the order in which the regionsare merged will alter the configuration of the resulting regionsin the final segmentation. This is true irrespective of the factthat the merging criteria remain unchanged. Thus, this aspectof setting strategy is an important one, and in fact, it is re-sponsible for many of the errors in the results of segmentingnatural scenes using existing segmentation systems.We have studied the problem of selecting the order in which

a low level system should process an image. The inclusion offocus of attention areas in the data structure is significant[13], [9] . These areas guide the system to parts of the imagethat should be processed first. The FOCUS OF ATTENTIONprocess selects the appropriate area by using a decision makingprocess that is based on the properties of the different areas.Once this is done, the SUPERVISOR determines the method ofdata selection within the area by assigning priorities to theFOCUS OF ATTENTION rules. Furthermore, this data selectionstrategy is not a fixed one. It varies from one area to the next,according to the properties of the data within each area. Wethus introduce the notion of a dynamic data selection methodin processing an image. The FOCUS OF ATTENTION processwill then execute the strategy that is now embodied in its rulesto choose specific regions and lines within the selected area.In a more general sense, setting the strategy includes not

only specifying the method of data selection, but also establish-ing priorities among the processing criteria embodied in therules. Before any of the KNOWLEDGE MODULES can match itsrule set to the data, it must first be selected. Obviously, thisrequires establishing a priority among the KNOWLEDGEMODULES. Furthermore, more than one rule within the sameset could be applicable at any time. When such conflicts arise,a priority ordering is needed to specify which one of a groupof rules should be selected to execute its actions. The SUPER-

157


VISOR determines these priorities, as well as the priority amongthe KNOWLEDGE MODULES. Unlike previous rule based sys-tems, the conflict resolution mechanism used is a dynamic oneas well. The order in which the rules are matched can actuallybe varied, based on the characteristics of the data. This ideaof a dynamic data driven iule ordering methodology is new,and improves both the efficiency of computation and thequality of the output [11] .The introduction of a dynamic element to both aspects of

strategy described above is possible mainly because of the crea-tion of a set of segmentation measures for region and line pro-cessing. These are computed for each area, stored in the STM,and are thus available to all of the processing modules. Theyprovide an implicit bottom-up feedback path, which influencesstrategy setting in each area. The SUPERVISOR uses this in-formation to compute the priorities of the KNOWLEDGEMODULES, the priorities of the rules within each module, andthe priorities of the FOCUS OF ATTENTION rules. In doing so,it defines both a conflict resolution strategy and a data selec-tion strategy. These constitute the top-down control paths ofthe strategy setting loop.The details of this dynamic strategy theory have been pre-

sented in [10], [1 1,and [16]. In this paper, we introduce theset of segmentation measures which serve as performanceparameters for the system. These measures collectively takeinto account the factors involved in defining a segmentation.They have the additional property of being dynamic, in thesense that they reflect the state of the segmentation at anypoint in time. When evaluated over each area of attention,they constitute a performance vector whose utility in deter-mining local and global strategies was described in [11], [16] .

The components of this vector are discussed in detail in thefollowing sections.

III. REGION UNIFORMITY

The uniformity of a feature over a region is inversely propor-tional to the variance of the values of that feature evaluated atevery pixel belonging to that region. A nil value for the vari-ance requires all pixels to have the exact same value for thefeature, while a large variance would indicate a large spreadfrom the mean of the feature value across the region. Let thevalue of feature F at each pixel i belonging to region R1 be fi,then the mean of these values fj is given by

fi Z fiAj (1)i ERj

where Ai is the area of region Rj. The variance of feature Fover region R, is then given by

a>= L (f- fj)2/Aj. (2)i ERj

The uniformity measure of an area a in the region map is de-fined as

U=1-1 WINj)/ (3)\Rj E- oe

where w; is the weight associated with the contribution of re-gion Rj to the measure, and the summation is taken over all

regions in the given area. N is a normalization factor designedto limit the maximum value of the measure to one

N= (I wJ (fmax fmin)2/2Rj Eal

(4)

where fmax and fmri are the maximum and minimum featurevalues over the area, respectively. This corresponds to theworst case condition for the measure, when all regions havethe maximum variance

2 (fmax - fmin )2Vmax 2 (5)

Since uniformity is basically a property that is increasinglynoticeable with size, the contribution of each region to theuniformity of the whole area is taken to be proportional to thesize of that region. The weight w; for region Rj will thus beequal to the area of the region Aj. Substituting (2) and (4) in(3), we obtain the following for the region uniformity measureof an area:

Ua, = 2 f )2 ( S Aj (fmaxfmjn)2-RjEa iE Rj /R U al

(6)The measure is seen to be a global one that represents all theregions in the area. However, each region's contribution to themeasure depends only on the features of that particular region,and can, in fact, be computed separately. We can write

UOT =1 - E: U ,Rj E ae

(7)

where u1a is the contribution of region R1 to the measure, givenby

uj= 2 , (fi-i) /i E Rj k E-k°a

Ak) (fmax fmin)2.

(8)

This can be simply written as

ujx= AjG, /AUaUaxl (9)

where A. is the sum of the areas of the regions in area ai.This renders the measure easy to update whenever regions

change due to splitting or merging. The efficiency of the com-putation is evident, in that we need only recompute the con-tributions of the regions that undergo a change; the contribu-tions of other regions remain constant. In case of merging tworegions, for example, the contribution of each region beforemerging is first subtracted from the measure, and the contribu-tion of the resulting merged region is then added. This is animportant property, since the measure must be updated afterevery modification of the regions.

IV. REGION CONTRASTThe previous measure takes into account the nonuniformity

inside each region. Contrast can be computed on the basis ofthe average values of features of adjacent regions. The assump-tion here is one of independence. Uniformity is computed asa measure within each region independent of the surrounding

1 5 8


1000

0 4

100 F>

zp 0

LL Qnf r-

-M<s O

z v

-,

LI0.3

0.2

10

0.1

0.1

0

1 10 100

SPATIAL FREQUENCY (Cycles/degree)

Fig. 3. Example of contrast sensitivity curve (from [31).

regions, while contrast is computed between adjacent regionsassuming that they have a uniform feature value equal to theaverage of that feature. The contrast between two regions Riand Rj is thus given by

Ifi -fj~Ci1 = I + t i = c1. (10)

This is seen to have a minimum value of zero and a maximumvalue of one. The contrast measure for a region is equal to aweighted sum of the contrast of that region with all of its ad-jacent neighbors. Each neighboring region Ri contributes tothe summation with a value that is proportional to the ad-jacency between it and the region Rj for which the measureis computed. Let this adjacency value be pij as defined in[13]; the measure for region R1 is then given by

cj= 1:Pijcii- 11AdjRi

Note that the summation of the adjacency values pij for allregions Ri adjacent to Ri is equal to one. The bounds of themeasure thus remain unchanged.In order to obtain a single contrast measure for the whole

area, we need to compute a weighted sum of the contributionsof each region in that area. In general, the contrast measurefor area ae is given by

Ca= E VjC1j / Vj, (12)Rj (=-ei / Rj E_=

where vj is the weight assigned to region Rj.As in the case of the uniformity measure, the weight assigned

to the contribution of each region to the contrast measuredepends on its size. The manner by which the weights varywith size, however, is not a linear one. Since the goal of thesemeasures is to "imitate" human judgment, let us consider howthe human visual system deals with contrast. Experimentsthat study the effect of spatial frequency on our perception ofcontrast yield what is known as contrast sensitivity curves [3] .These are plots of the lowest contrast value between stripes ofa certain spatial frequency, that enable humans to distinguishbetween these stripes as a function of the spatial frequency.An example of such a plot for a fixed average intensity level isshown in Fig. 3. The sensitivity decreases for very high and

7 -2c - 0 ar 2o- 3a

REGION AREA

Fig. 4. Weights as a function of region size.

very low spatial frequencies and has an optimal value at a fre-quency that corresponds to approximately three degrees ofvisual angle.Applying the above observations to the contrast between re-

gions in a segmented image, we can argue that there is an op-timal region size for which contrast between regions visuallyaffects the results of segmentation the most. This is because,as the spatial frequency increases (the size of the region de-creases), the perception of contrast is enhanced until we reachthe optimal size. As the size of regions decreases beyond that,contrast perception deteriorates because of the limitations im-posed by visual acuity. The curve shown in Fig. 4 representsthe weights given to regions as a function of region area, in theform of a normal distribution.This was chosen as a crude, but simple approximation of a

single valued function that simulates the behavior describedabove. The mean of the curve corresponds to the optimal re-gion size, and the standard deviation is chosen so that theweights will approach zero at the limits of visual acuity (thiscorresponds to the image resolution in computer vision). Theweight for region R1 is thus given by

V>=i exp [(A 2 (13)

where p and a are the mean and standard deviations of thecurve, respectively.The previous two measures account for the segmentation of

an image into regions that are uniform and distinct. They areseen to be complementary. Both measures must be high toindicate a good segmentation. A composite measure of seg-mentation must take this into account. In the following twosections, we discuss the effect of lines on the evaluation of asegmentation, by presenting two appropriate measures.

V. LINE CONTRASTThis measure is designed to check the validity of existing

lines in an image segmentation. We infer the presence of a linein an image if one of two situations occurs.

1) There exists an appropriate amount of contrast in thefeatures of regions on either side of the line.2) There is enough evidence, in terms of a local gradient in the

features of pixels belonging to the line, to justify its presence.

159

IEEEL! TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. PAMI-7, NO. 2. MARCH 1985

The first condition accounts for most of the lines that hu-mans infer in an image. The second condition is usually satis-fied in conjunction with the first. It is included separately toaccount for certain special cases where lines exist as separateentities, that are independent of the regions that surroundthem. An example of this occurs when a thin object, whosewidth is the same as that of a line, occludes a uniform objectin the background. Since the regions on both sides of the linebelong to the same object, there is virtually no contrast be-tween them. Still there is a strong local gradient brought aboutby the existence of the thin object.The measure defined here is a composite one, in order to ac-

count for both types of lines cited above. The first category isbest represented by the contrast between regions on both sidesof a line. On the other hand, a local gradient measure will bemore useful for lines belonging to the second category. Thesecontribute weakly to a contrast measure even when they arerightfully in place. Let the contrast of a feature F across lineL be dj then

dj- FF ' (14)1+ I-.F

where the subscripts I and r refer to the left and right regions,respectively,

F Z= E pil i,Ri

Ir X Pkr f k,Rk

and pi, is the adjacency of a region Ri to the left of the line,fi is the average value of feature F over that region, Pkr is theadjacency of a region Rk to the right of the line,fk is the aver-age feature value over that region, and the summations are takenover all regions to the left and right of the line, respectively.For the second category, we define a measure of the local

gradient across line Lj as gj, where

gi =_ Gj1 6)(fmax fn(in)

Gj is the average gradient of feature F over the pixels belongingto line Lj, fmax and fin are the maximum and minimum fea-ture values in the area as defined in Section III.The contrast measure given in (1 4) accounts for the first

category of lines, which includes most of the lines in an image.The second group of lines is more adequately represented bythe local gradient measure of (16). The latter are characterizedby a high value of gj coupled with a low value of dj. A hybridmeasure hi can thus be constructed for the contrast across aline Li

gj if gj > 3dj and dj < e;(17)

d/j otherwise,

and e is a constant near zero that indicates very low contrast.A line will thus contribute to the contrast measure through theproperties of the regions on both sides of it. If, however, theregion contrast is low, the local gradient across the line will bemore representative of its contribution.A composite measure of line contrast over an area a can now

be computed from the individual line measures as follows:

H = 53 whvjhj / ,Li E a / LG a

(18)

where wj is the weight assigned to the contribution of line Li,and is taken to be equal to the length of the line. The summa-tion is over all lines in the area.

VI. LINE CONNECTIVITYThe previous measure only accounts for the validity of lines

present in the image segmentation. A complete measure ofsegmentation should also involve lines that should have beenpresent, but are in fact missing. This kind of error in segmenta-tion is more difficult to quantify, although it is easily detectedby a human observer. If a process is able to observe the absenceof lines, it would not have missed them in the first place. Never-theless, some inference can be made about missing lines. Agood clue to this is the presence of incomplete lines. By theirnature, lines in an image tend to close on themselves, or onone another. This is also in accordance with one of the prin-ciples of Gestalt psychology, that of good continuity [7].Lines rarely stop suddenly and disappear, or have an open endthat goes nowhere.Based on this argument, the measure developed here is one

of connectivity of lines. The end of a line is said to be closedif it terminates on another line, or if it closes on itself. A lineis closed if both its ends are closed. The number of lines thathave one or both ends open is an indication of the degree ofdiscontinuity in the lines. Let tj be the number of closed endsof line Li and Ij be the length of that line. Then

Tee £ Ijtj/2 E ljLi Eat Li E a

(19)

will represent a global measure of the connectivity of all thelines in area a of the line map. Note that t1 is a tri-valuedquantity (0, 1, 2,), and hence the measure Tog is bounded be-tween the values of zero and one. A low value of Tof would in-dicate that there is a large amount of discontinuity in the lines,because of the large number that have their ends open.Although the measure described here does not account for

all the lines missing from an image segmentation, it does go along way in doing so. This is because lines in an image tend tobe partially present, rather than totally missing. It is the totallymissing lines that are not represented in the measure. Again,the connectivity measure complements the contrast measurefor lines, as is the case with the uniformity and contrast mea-sures for regions.In order to complete our evaluation of an image segmenta-

tion, other performance parameters are added to the abovefour measures, and are designed to account for the presence oftexture in an area of the image.

VII. TEXTURE MEASURES

When we refer to region uniformity, we usually mean uni-formity over grey level features. Thus, uniform regions arethose that do not contain any texture. However, a more gen-eral interpretation of uniformity would include texture fea-tures as well. If we extend this to the definition of segmenta-

160


tion given at the beginning of this section, the regions in anoutput segmentation would be required to be uniform in tex-ture features. This means that textured regions would not bedivided into the grey level uniform subregions representing theunits of the textural pattern. Instead, they would be groupedinto single entities, despite the fact that their grey level uni-formity would be quite low. One of the main problems withthis approach is that textural features are inherently global fea-tures that must be computed over an area in the image. Someapproaches to texture analysis assign local values to these fea-tures at every pixel in the image [8]. This is done by comput-ing the global measure over an area surrounding each pixel.The choice of that area is what has become known as the win-dow size problem [19] . A large window is likely to include arange of values at the boundaries between textured regions. Asmall window, on the other hand, will be more sensitive tolocal variations due to the individual patterns of a texture. Ef-forts to determine an optimal window size have had limitedsuccess [20], and an exhaustive search would add to the com-putational burden.Our approach to the problem is quite different. Regions are

defined to be uniform over grey level features. The definitionof segmentation given in the introduction is thus applicable,and so are the measures described in Sections III and IV. Tex-ture areas are then defined explicitly, and are represented as agroup of uniform grey level regions that correspond to the prim-itive elenments of texture [14]. These regions are usuallysmall in size, and exist in clusters of relatively large numbers.Texture properties can now be computed by averaging the fea-ture values over the regions. A histogram of the average inten-sity of all the regions belonging to the same area, for example,would be multimodal. This is because each group of regions(for example, background and foreground) will contribute adistinctive intensity value, and hence will produce a separatepeak in the histogram. Thus, the elementary regions replacethe windows that are normally required for computing averagetextural features, and local values at every pixel in the imageare no longer necessary. This solves the window size problem.A global measure over the regions that fall within an area willindicate whether they indeed belong to the same texture. Thecomputational efficiency is also seen to improve drastically.Each focus of attention area has a label that classifies it as

being "textured," "smooth," or as being "bounded" by longconnected lines. The four performance measures describedabove are computed for each area. It is evident that the rela-tive importance of each measure is dependent on the type ofarea for which it is used. Region uniformity is less importantin textured areas than in smooth ones. Connectivity is impor-tant for textured and bounded areas, contrast is important insmooth and textured areas, although more so for regions thanlines. Thus, the preceding measures do not have the sameweight in different areas. This suggests the inclusion of thearea type Q,, deduced by the system, as a performance param-eter for area oa. The latter can then be used to emphasize theappropriate subset of measures in a particular area.Other important properties of an area are the number of re-

gions and the number of lines it contains. In addition to beingclues for texture, as performance parameters they have a large

influence on setting strategy [10]. These numbers are norma-lized over the areas by computing the number of regions (andlines) per unit area for the whole image and comparing it withthe same quantity for each area. Consider the value

R =NROJAo NR '-ATa NRT/AT NRT AJ' (20)

where NRo, and NRT are the number of regions in area al andin the whole image, respectively. If Rc, is greater than unity,the density of regions in a will be greater than the average den-sity for the whole image, and vice versa. A value in the range[0, 1 ] can then be computed by dividing by a multiple of theaverage density (e.g., 3), and truncating any higher values.Similarly for lines, we have

NLc,a/A , NLoc 'ATa NLTIAT NLT-Ac,

(21)

The performance vector for each area in the image will thusinclude components for region uniformity, region contrast,line contrast, line connectivity, number of regions, and numberof lines. Note that the first three components are computedper feature in the image. For black and white images, eachcomponent can be computed for the grey level intensity ofregions and lines. For color images, each measure will con-tribute three elements to the performance vector, one for eachcolor channel. In general, the uniformity and contrast of otherfeatures of regions and lines can be added to the performancevector. For m such features, the performance vector P., forarea a will haveM = 3m + 3 components, and is given by

PO := [Ul .

* * UO , ci ...

*** cam,eH1 ,Hm, Tc,, Ro, Uo, (22)

where

Ul to UOg are m region uniformity measures for area a,C, to CT7 are ni region contrast measures for area a,H' to Hoe are ni line contrast measures for area a,T,, is the line connectivity measure for area at,Rc, is a normalized value for the number of regions in a,

and

Lc, is a normalized value for the number of lines in a.

Each area cr will be represented in the strategy determinationprocess by its performance vector P., and its type TO',.

VIII. RESULTS AND CONCLUSIONS

Fig. 5 shows a picture of a 256 X 256 outdoor scene.1 Theboundaries of regions in a segmentation of the image are shownin Fig. 6, and in Fig. 7 the lines detected in the image are given.The smooth areas that are computed based on these segmenta-tions are shown in Fig. 8. They are seen to correspond to thesky, the wall, and the grassy areas.The measured parameters for the sky area are given by Table

I. It is seen that the area is characterized by a high uniformityof the three color features, also evident from the low value of

1The scene was provided by E. Riseman of the University of Mas-sachusetts in Amherst.

161

IEEE TRANSACTIONS ON P'ATTERN ANALYSIS ANI) MACHINE INTELLIGENCEI, VOL. PAMI-7, NO. 2, MARCH 1985

Fig. 5. The ihage of an outdoor scene.

Fig. 7. Image segmented by lines.

Fig. 6. Image segmented into regions.

the variances. The normalized number of regions is low, sincethe area is composed of a small number of large regions. Thepresence of lines in this smooth area should be noted. The con-trast between these lines is low, and so is the contrast betweenregions. This is an indication that the area requires more linedeletion and region merging.Three textured areas are shown in Fig. 9. The measurements

for the large tree are given in Table 11. The variance of thearea is seen to be high. The uniformity within each region islower than for smooth areas, but the number of regions ismuch greater (which also reflects the smaller size of these re-gions). The number of lines is also high, and their connectivityis large. Region and line contrast are seen to exceed the valuesfor smooth areas. The diversity in the values of the parametersfor areas 15 and 9 underlines their effectiveness in dynamicallysetting the processing strategy.Other experiinents were conducted to demonstrate the dy-

namic characteristics of these segmentation measures. Starting

Fig. 8. Examples o0 smooth areas.

IABLE IINFORMATION ON AREA 15

Area TypeBounding RectangleArea SizeTotal Line LengthsNumber of RegionsNumber of LinesCurrent RegionCurrent Line

Area VariancesRegion UniformityRegion ContrastLine Contrast

Line Connectivity

SMOOTH(38, 165) & (255, 256)10620

4046

481327

Red Green

19.55 15.690.979 0.9860.222 0.2260.108 0.115

0.6125

Blue

29.820.9420.320.146

162


Fig. 9. Examples of textured areas.

TABLE 11INFORMATION ON AREA 9

Area TypeBounding RectangleArea SizeTotal Line LengthsNumber of RegionsNumber of LinesCurrent RegionCurrent Line

Area VariancesRegion UniformityRegion ContrastLine Contrast

Line Connectivity

TEXTURE(2, 128) & (94, 251)83938749659

241174

Red

121.060.520.3610.313

0.826

Green

159.910.2880.4390.354

Blue

124.010.5190.4870.421

with an arbitrary segmentation of the image, rules were appliedto merge and split regions, as well as add, delete, and join lines.Changes in the measures after each single action were recordedand plotted against the number of actions. Curves for regionuniformity, region contrast, and line connectivity show a de-caying rise in values that is followed by asymptotic saturation.The system was able to perform the actions that produced

larger improvements in the performance measures before otherless effective actions. This is because the rules are ordered ac-

cording to their influence in improving the measures. In addi-tion, the FOCUS OF ATTENTION module executes a strategythat prefers regions and lines that produce larger increases inparameter values. Less important actions could thus be com-

pletely left out, because they only occur near the end of pro-

cessing where the output is already very close to the final re-

sult. This indicates the usefulness of these measures in settinga dynamic strategy for the system.The curve for line contrast behaved in a different manner. It

remained constant or even showed a slight decrease in valuewith the increase in the number of actions performed. This isbecause the initial configuration of lines usually corresponds

to the high contrast local discontinuities in the image array.The rules tend to join lines together at the expense of addingpoints that do not possess high contrast characteristics. Theinitially high overall contrast value for lines will thus suffer aslight decrease in value. This suggests that although line con-trast is effective in the static distinction between different areas,it is not suitable for dynamic analysis.In Section II we discussed the various factors involved in

setting the strategy for a rule based low level image segmenta-ion system. The details have been presented elsewhere [91-[111 . In this paper, we have defined a set of parameters thatmeasure the performance of the system as reflected by the stateof the image output at any point during processing. A pool offocus of attention areas is generated according to certain cri-teria embedded in rules that are designed for this purpose. Per-formance parameters are subsequently computed for each area,based on the initial configuration of regions and lines withinthat area. These parameters are stored in the STM, and aredynamically updated during processing so that they can con-tinuously represent each area at any point in time. The FOCUSOF ATTENTION process selects the most appropriate area towork on at any time. The criteria used are those of processingneed and uniform time sharing. Once an area is selected, theSUPERVISOR proceeds to define the strategy within that area.It evaluates decision functions that reflect the effect of theperformance parameters of the area on a number of strategycomponents.The key factor in creating a dynamic strategy is the ability

to dynamically measure the state of the segmentation, andconsequently obtain an indication of the distance from thefinal goal. The measure of segmentation presented here iscomposed of four components that are integrated into theperformance vector of each area of attention. These dynamicparameters depend only on the input image and the state ofthe output, as reflected in the segmented image. Consequently,the measure devised here is a general purpose, context inde-pendent one. It is also an on-line measure that can be efficientlyupdated during processing. Results of evaluating the compo-nents of the measure for different areas show that it provides aconsistent means of reflecting the properties of various areasin the image. In addition, the dynamic behavior of some com-ponents is useful in providing the means for adjusting the pro-cessing strategy to achieve efficient processing.

REFERENCES[1l I. E. Abdou and W. K. Pratt, "Quantitative design and evaluation

of enhancement/thresholding edge detectors," Proc. IEEE, vol.67, pp. 753-763, May 1979.

[2] D. J. Bryant and D. W. Bouldin, "Evaluation of edge operatorsusing relative and absolute grading," in Proc. IEEE Comp. Soc.Conf. Pattern Recognition Inage Processing, Chicago, IL, Aug.1979, pp. 138-145.

[31 F. W. Campbell and J. G. Robson, "Application of Fourier anal-ysis to the visibility of gratings," J. Physiol., London, vol. 197,pp. 551-566, 1968.

[41 G. B. Coleman, "Image segmentation by clustering," Image Pro-cessing Inst., Univ. Southern Cal., Los Angeles, USCIPI Tech.Rep. 750, July 1977.

[51 J. R. Fram and E. S. Deutsch, "On the quantitative evaluation ofedge detection schemes and their comparison with human per-formance," IEEE 7tans. Comput., vol. C-24, pp. 616-628, June1975.

163

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. I'AMI-7, NO. 2, MARCH 1985

161 L. Kitchen and A. Rosenfeld, "Edge evaluation using local edgecoherence," IEEE Phans. Syst., Man, Cybern., vol. SMC-1 1, pp.597-605. Sept. 1981.

[71 K. Koffka, Principles of Gestalt Psychology. New York: Har-court, 1963.

[81 M. D. Levine, "Region analysis with a pyramid data structure,"in Structured Computer Vision: Machine Perception ThroughMierarchical Computation Structures, S. Tanimoto and A. Klinger,Eds. 1980, pp. 57-100.

[91 M. D. Levine and A. Nazif, "An experimental rule based systemfor testing low level segmentation strategies," in Multicomputersand Image Processing: Algorithms and Programs, K. Preston andL. Uhr, Eds. New York: Academic, 1982, pp. 149-160.

[101 - -, "Performance measurement and strategy evaluation for rulebased image segmentation," Dep. Ilec. Eng., Comput. VisionGraphics lab., McGill Univ., Montreal, P.Q., Canada, TR82-1,Mar. 1982b.

[111 -, "Rule-based image segmentation: A dynamic control strategyapproach," Comput. Vision Dep. Elec. Eng. Comput. Vision Ro-botics Lab., McGill Univ., Montreal, P.Q., Canada, TR-83-9,June 1983b; also Comput. Vision, Graphics, Image Processing, tobe published.

[121 --, "An optimal set of image segmentation rules," PatternRecognition Lett., vol. 2, pp. 243-248, June 1984.

[131 M. D. Levine and S. l. Shaheen, "A modular computer vision sys-tem for picture segmentation and interpretation," IEEE 7rans.Pattern Anal. Machine Intell., vol. PAMI-3, pp. 540-556, Sept.1981.

[14] T. Matsuyama, K. Saburi, and M. Nagao, "A structural analyzerfor regularly arranged textures," Comput. Graphics Image Pro-cessing, vol. 18, pp. 259-278, Mar. 1982.

[15] A. Nazif and M. D. Levine, "A rule based image segmentationsystem," presented at 7th Conf. Canadian Soc. Man-Comput.Studies, Waterloo, Ont., Canada, May 10-12, 1981.

[16] --, "Setting strategy in a rule based vision system," presented atProc. 4th tnt. Conf. Canadian Soc. Comput. Studies Intell.,Saskatoon, Sask., Canada, May 17-19, 1982.

[17] --, "Low level image segmentation: An expert system," IEEETrans. Pattern Anal. Machine Intell., vol. PAMI-6, pp. 555-577,Sept. 1984.

[18] W. K. Pratt, Digital Image Processing. New York: Wiley, 1978,pp.495-501.

[191 E. M. Riseman and M. A. Arbib, "Survey: Computational tech-niques in the visual segmnentation of static scenes," Comput.Graphics Image Processing, vol. 6, pp. 221-276,1977.

[201 A. Rosenfeld and M. Thurston, "Edge and curve detection forvisual scene analysis," IEEE Trans. Comput., vol. C-20, pp. 562-569, May 1971.

[211 W. A. Yasnoff, J. K. Mui, and J. W. Bacus, "Error measures forscene segmcntation," Pattern Recognition, vol. 9, pp. 217-231, 1977.

Martin D. Levine (S'59-M'66-SM'74) was bornin Montreal, P.Q., Canada, on March 30, 1938.He received the B.Eng. and M.Eng. degrees inelectrical engineering in 1960 and 1963, respec-tively, from McGill University, Montreal, andthe Ph.D. degree in electrical engineering in1965 from the Imperial College of Science andTeclnology, University of London, London,England.He is currently a Professor of Electrical En-

gineering in the Department of Electrical En-gineering, McGill University. During 1972-1973 he was a member ofthe Technical Staff at the Image Processing Laboratory of the Jet Pro-pulsion Laboratory, Pasadena, CA. During the 1979-1980 academicsession, he was a Visiting Professor in the Department of ComputerScience, Hebrew University, Jerusalem, Israel. His research interestsare computer vision, biomedical image processing, and robotics. He hasauthored the book Vision in Man and Machimie (New York: McGraw-Hill, to be published), and has coauthored, with P. Noble, the bookComputer Assisted A nal .ses of Cell Locomotion and Ci'emotaxis (CRCPress, to be published).Dr. Levine is an Associate Editor of Computer Vision, Graphics and

Image Processing, and was the General Chairman of the Seventh Inter-national Conference on Pattern Recognition held in Montreal duringthe summer of 1983. Recently, he was appointed a Senior Fellow inthe Canadian Institute of Advanced Research.

Ahmed M. Nazif was born in Cairo, Egypt, inJuly 1952. lie received the B.Sc. and M.Sc. de-grees in electrical engineering from Cairo Uni-versity in 1973 and 1975, respectively, and thePh.D. degree from McGill University, Montreal,P.Q., Canada, in 1983.From 1973 to 1976 he was a Demonstrator at

Cairo University. From 1976 to 1983 he was aResearch Assistant at the Computer Vision andRobotics Laboratory of McGill University.Since July 1983, he has been an Assistant Pro-

fessor at the Department of Electrical Engineering, Cairo University.

164

dynamic measurement of computer generated image segmentations

Documents