performance evaluation of grouping algorithms

Performance Evaluationof Grouping Algorithms

Vida MovahediElder Lab - Centre for Vision Research

York University

Spring 2009

Centre for Vision Research, York University 2

Overview Grouping and evaluation methods

Region-based measures

Boundary-based measures

Mixed measures

Alignment measure





Mixed measures

Alignment measure


Grouping Edge segments: Example


Perceptual Organization/ Grouping

A process of assembling features into groups which are perceptually significant based on various cues (Lowe, 1985)

The problem of aggregating primitive image features that project from a common structure in the visual scene (Elder, 2002)


Evaluation Measure How good is each grouping?

Which algorithm has a better performance?

What is the best grouping that can be achieved?

Note differences with regional segmentation evaluation


Evaluation Methods Three main categories (Zhang, 1996)

Analytical methodsConsider the algorithms themselves, e.g. based on the a priori knowledge they use (not based on output of the algorithms)

Empirical goodness methodsBased on the outputs of the algorithms, e.g. based on the intra-region uniformity of the segments, or the inter-region contrast between the segments.

Empirical discrepancy methodsA reference segmentation or ground truth is assumed, to compare the outputs with


Goal: Measure Discrepancy


SOD: Salient Object Dataset Based on Berkeley Segmentation Dataset

(BSD)

300 images

7 subjects

1

1

1

1

1





Mixed measures

Alignment measure


Region-based Discrepancy (Young, 2005),(Ge, 2006), (Goldmann, 2008)

A and B two boundaries

RB the region corresponding to a boundary B and |RB| the area of this region

1: maximum discrepancy,

0: maximum consistency

BA

BAAB RR

RRError

1


Interpretation

BA

BAB

BA

BAAAB RR

RRR

RR

RRRError

||||


Evaluation by this measure

Not sensitive to spikes, wiggles, shape

>=

(more error)


Examples of near-optimal cases





Mixed measures

Alignment measure


• Distance of one point a from B

• Distance Signature of all a in A

• One directional Hausdorff

• Two directional Hausdorff

Boundary-based Distance

),(,),(),( abdMinMaxbadMinMaxMaxBAHAaBbBbAa

}),({),( AaadBASD BB

),(min)( badadBb

B

)(max)),(max(),( adBASDBAh BAa

B

),(),,(max),( ABhBAhBAH



Not sensitive to wiggles, shape

Not sensitive to the distance distribution, but only to the maximum value


Geodesic Distance: the min. distance between two points a and b without cross

Euclidean vs. Geodesic Distance

Euclidean Distance: the min. distance between two points a and b

),(min),( ieAa

e abdAbdi

),(min),( igAa

g abdAbdi



Almost the same

by De

Almost the same by De & Dg





Mixed measures

Alignment measure


A mixture of boundary-based and region-based Penalizing the over-detected and under-detected

regions by their Euclidean or Geodesic distances

fpfn N

kkB

fp

N

jjA

fn

qdN

pdN

BAD11

)(1

)(1

2

1),(

pj, j=1..Nfp are pixels in the false negative region (RB-RA)

qk, k=1..Nfn are pixels in the false positive region (RA-RB)


Evaluation by this measure Not penalizing effectively, e.g. narrow false

positives below


Correspondence Problem The false negative and false positive regions can be

very small, yet the boundaries be very different

Segments on one boundary should correspond to segments on the other


Alignment The order of matching points on the two boundaries

should be monotonically non-decreasing.

nmbaMatchbaMatchji njmi )( and ,)( , If

)(),( :sother wordin

B,,,),(minarg)(

*

**

adbad

bAabadaMatchb

B

Bb


Correspondence (Cont.) Note that if correspondence is maintained,

De will work almost like Dg!





Mixed measures

Alignment measure


Alignment Distance Main idea: We need to find the ‘alignment’ that leads to

minimum total distance.

Method: Use N samples on each boundary (equally spaced) Find the NxN matrix of Euclidean distances. The diagonals show correspondences with some rotations The one with min sum of distances is the best

correspondence and its sum is our measure of discrepancy.


Alignment Measure (cont.)

4321

4321

4321

4321

4321

4321

4321

4321

432 14 3 2 1

dddd

cccc

bbbb

aaaa

dddd

cccc

bbbb

aaaa

dddd

dddd

dddd

dddd

dddd

dddd

dddd

dddd

d

c

b

a

Note: Order of both samples increases clockwise

Nirotii

NrotbadBAD

..11..0

),(min),(


Evaluation by this simple measure Samples falling out of phase

Solution: finer sampling on one boundary

Reference handgrouping with error=1177.8474

A better curve with less error=872.5259

>=

(more error)


Bimorphism (Tagare, 2002)

A method to let correspondence of 1 to many and many to 1 symmetric


A symmetric Alignment Distance Edit cost of changing one string to another

Edit operation, cost of operation

A sequence of operations taking A to B

Symmetric:

naaaA ....21

mbbbB ...21

bas : edit ofcost :)(s),()( :choose we badba

ksssS ...21

k

iisS

1

)()(

BASSBA to takingsequenceedit an is |)(min:),(

A)(B,B)(A, then ,)()( If abba


Example

4321 aaaaA

321 bbbB

),,

,,(

342423

1211

bababa

babaS

)3,4(),2,4(),2,3(),1,2(),1,1(T

5|| T

),(),(),(),(),()( 3424231211 badbadbadbadbadS

)(),( SBA


Cyclic shifts Cyclic shifts

Alignment Distance

Dynamic programming

Complexity:

(Maes, 1990) Complexity:

])[],([

||

1),( BA

TBADA

])[,(])[],([ BABA

mlnkBABA lk 0,0|)(),(min:])[],([

AAnkaaaaaaa knknk )( and ,1,......)...( 0

1121

).( 2 nmO

)log.( mmnO


Examples

Alignment Distance=7.73


Examples

Alignment Distance=3.25


Evaluation by this measure Note: If using Euclidean distance, there is

no sensitivity to region elite curve with HD8-Mae


References

(Elder, 2002) J. H. Elder and R. M. Goldberg (2002), "Ecological statistics of Gestalt laws for the perceptual organization of contours." J Vis, vol. 2, pp. 324-353.

(Zhang, 1996) Y. J. Zhang. (1996), “A survey on evaluation methods for image segmentation”, Pattern recognition 29(8), pp. 1335.

(BSD) D. Martin (2001), "A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics," Proceedings of the 8th IEEE International Conference on Computer Vision, vol. 2, pp. 416-423.

(Ge, 2006) F. Ge, S. Wang and T. Liu (2006), "Image-Segmentation Evaluation From the Perspective of Salient Object Extraction," Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1, pp. 1146-1153.

(Goldmann, 2008) L. Goldmann. (2008), Towards fully automatic image segmentation evaluation. Lecture notes in computer science 5259 LNCS, pp. 566.

(Young, 2005) D. P. Young (2005), "PETS Metrics: On-line performance evaluation service," Proceedings - 2nd Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, VS-PETS, vol. 2005, pp. 317, 2005.

(Huttenlocher, 1993) D. P. Huttenlocher (1993), “Comparing images using the Hausdorff distance”, IEEE transactions on pattern analysis and machine intelligence 15(9), pp. 850.

(Tagare, 2002) H. D. Tagare. (2002), “Non-rigid shape comparison of plane curves in images”, Journal of mathematical imaging and vision 16(1), pp. 57.

(Maes, 1990) M. Maes (1990), “On a cyclic string-to-string correction problem”, Information processing letters 35(2), pp. 73.

performance evaluation of grouping algorithms

Documents