Hierarchical Region-Based Segmentation by Ratio-Contour

Jun Wang

April 28, 2004

Course Project of CSCE 790

a) What is image segmentation?Partition image (composed of pixels ) into several regions. Each region corresponds to meaningful structures or objects.

b) Why do image segmentation? Representing Information and knowledge efficiently;

Reduce the computation for further processingImportant step for high level computer vision problem, such as object recognition and event recognition.

C) Easy to do image segmentation?-----NO!Image is complicated: noise, texture, shadow, illumination, etc.Number of pixels is huge;Incorporation of prior knowledge;

Background: Introduction of Image Segmentation

Classification of Segmentation Methods

Low level methods Edge-based Region-based Clustering-analysis based Graph-theory approach

Middle level methods Active contour Level set methods

High level methods Model-based/Knowledge-guided Statistical-learning based method Template deformation

Graph Based Image Segmentation

Graph model of image

2D image ---- graph

pixels, points from edges ---- vertexes in graph

pixels pair or points pair ---- edges in graph

pixel similarity, connection between points

---- weight of edge

segmentation problem transfer to graph problem

image region segmentation ---- graph cut

boundary detection in image ---- path search in graph

Vv),( EVG

ijew

Eeij

Graph Based Image Segmentation Methods

Graph cut basedMinimum Cut (1993)

Ratio Region

Normalized Cut (1997)

Ratio Cut (2002)

Cycle search basedShortest path algorithm (1996)

Strongly connect component (2003)

Ratio Contour (2003)

Background: Introduction of Ratio Contour Algorithm

Objective

find Minimum Ratio Alternate (MRA) cycle in the graph, which is looked as the most salient boundary in the image.

Polynomial time computable solution for this cycle search problem.

Problem Formulation – obtain fragments image from original image

Original image Canny detection Fragments image

(after smoothing)

Problem Formulation – Construct a undirected graph for the smoothed fragments-image

are vertices constructed from the endpoints of the

fragments are edges consisting of two kinds of fragments:

real fragments and virtual fragments

),( EVG

},...,,{ 21 nvvvV },...,,{ 21 meeeE

Definition of weight

if is real fragment

if is virtual fragment

the curve from can be describe by function

is the curvature at point

is a constant related with the balance between smooth and

proximity

e

dttktew ))()(()( 2

1

0)(t

e

e

)(tk

)(tFe

e 10),( ttFe e

e

dtel )(

Cost function of the boundary

is a cycle in graph

Define the cost function as:

Optimize the cost function to obtain the most salient closed boundary with the minimum

),( EVG },...,,{ 21 neeeB

Be e

Be erc l

wBL )(

)(minarg BLB rcB

rcL

Problem solution – Ratio Contour algorithmReduction 1

Reduce the weight and length of solid edge to be zero by merging the weight and length of solid edge into the weight and length of adjacent dash edge.Reduction 2

Reduce the problem of finding a MRA cycle to the problem of finding a Negative total Weight Alternate (NWA) cycle.

Reduction 3Reduce the problem of finding a NWA cycle in the graph to the

problem of finding a Minimum-Weight Perfect Matching (MWPM) in the same graph.Summary

Finding WMPM cycle has classical polynomial time solution. And the reduction processes above are all polynomial time computable.

------ Details? Please refer to the reference.

Segmentation results on real images by Ratio Contour

Segmentation results on medical images by Ratio Contour

Further developmentsolid-dash edges based graph construction to points basedgraph construction -----why?

Fragments image Directed points image Directed points image (after smoothing) (with small length)

),(

EVG

Further developmentcurve length and curvature based weight definition to transition possibility based definition

the edge weight by calculating the transition possibility from vertex to .

by stochastic completion method. (Thornber and Williams 1996)

jiij vvw

iv

jv

ijp)log( ijij Pw

Further developmentundirected graph construction is developed in two step:

1. Construct directed graph based on previous weight calculation method.

2. Transfer directed graph to undirected graph by duplicating the vertex.

jiij PP

Boundary Based Segmentation

1. no region information is recorded;

2. the later detected boundary maybe traverse the form one;

3. may result in contradiction: same part belongs to different boundary inside area.

4. multi-boundary detection based segmentation can not be competent for hierarchical segmentation.

Example of Multi-boundary Detection

Region-based Segmentation

1.partition the graph after each iteration of boundary detection;

2. record the corresponding region information in Label Matrix;

3. the further boundary detection is executed on the generated multiple graphs, separately;

4. easy to extend for hierarchical segmentation.

Example of Region-based Segmentation

Example of Region-based Segmentation

Label Matrix – after three iteration of segmentation

Previous Example

When to stop? --- Threshold setting

Set a threshold of the total weight of the detected boundary. If the weight is larger than the threshold, stop segmentation.

Segmentation methods two kind of methods for the hierarchical segmentation:

Depth First Segmentation (DFS);

Broad First Segmentation (BFS);

The results are same.

Open boundary problemThe boundaries traverse the other region or reach the edge of the image. It is not a completely closed boundary.

How to update the algorithm for necessary open boundary detection?The current boundary detection algorithm is for

closed boundary, which must guarantee closure.

Boundary Vertex – special vertex in the graphEach extracted boundaries, include the original “boundary” - edge of image, are looked as a kind of complicated vertex. Each boundary is describe by a uniform vertex.

Property of Boundary Vertex1. have length and shape;2. every two points in the same boundary, the connection possibility is one. It means that the weight equals zero.

Illustration of Boundary VertexConnection between two vertex and traverse the former extracted cycle boundary.

How to calculate the weight between and ?

1p 2p

1p 2p

Calculating the weight as following.

transition possibility from to ,

based on the definition of weight

42341312 PPPP 1p

4213

42341342341312

loglog

logloglog

PP

PPPwwww

2p

Synthesized Experiments

Some Previous work (recently) S. Wang, T. Kubota, J. M. Siskind. Salient boundary detection using ratio contour, NIPS 2003

L.R. Williams & Thornber – using strongly connected component to extract boundary.

J. H. Elder & S. W. Zuker – using shortest path algorithm to computing contour closure.

Related Work (by Joachim Stahl )1. contour intersection detection;

2. branch & band for solving counter intersection;

Ongoing Work1. choose a efficient way for gap filling, especially large gap filling;

2. do more experiments on real images.

Some Reference 1) S. Wang, J. Wang, T. Kubota. From Fragments to Salient Closed Boundaries:

An In-Depth Study, to appear in CVPR, Washington, DC, 2004 2) S. Wang, T. Kubota, J. M. Siskind, J. Wang. Salient Closed Boundary Extract

ion with Ratio Contour, under review 3) J. Elder and S. Zucker. Computing contour closure. In ECCV 1996

4) L. Williams and K.K. Thornber. A comparison measures for detecting natural shapes in cluttered background. IJCV 2000.

5) L. Williams and D. Jacobs. Stochastic completion fields: A neural model of illusory contour shape and salience. Neural Computation, 1997

6) S. Wang, T. Kubota, J. M. Siskind. Salient boundary detection using ratio contour, NIPS 2003

7) S. Mahamud, L.R. Williams, K. K. Thornber and K.Xu. Segmentation of multiple salient closed contours from real images. IEEE Transaction on PAMI, 2003.8) Course notes of CSCE 790

Thank You!