Background Removal of Multiview Imagesby Learning Shape Priors

Yu-Pao Tsai, Cheng-Hung Ko, Yi-Ping Hung, and Zen-Chung Shih

ABSTRACTABSTRACT

Image-based rendering has been successfully used to display 3-D objects for many applications. A well-known example is the object movie, which is an image-based 3-D object composed of a collection of 2-D images taken from many different viewpoints of a 3-D object.

INTRODUCTIONINTRODUCTION

MVI ( Multiview Image)MVI ( Multiview Image)

Ο ( omicron )Θ ( thet ) : pan angle Φ ( phi ) : tilt angleMVI has three basic characteristics:

1) When an equi-tilt set of the MVI is captured, a large proportion of the background scene is static.2) Only one interesting object is presented in every image of the MVI.3) The foreground and background color distributions are distinct in most cases.

Proposed Flowchart Proposed Flowchart

The proposed approach aims to let every single image segmentation,rather than only those in neighboring viewing directions

To take the shape prior into account, the user is required to selectd a subset of acceptably segmented images. The 3-D shape is then generatedfrom these selected images.

AUTOMATIC INITIAL AUTOMATIC INITIAL SEGMENTATION :SEGMENTATION :

GraphGraph Cut Image Cut Image SegmentationSegmentation Graph cut image segmentation requires the user to

interactively mark some pixels as being inside the foreground objects, and others as a part of the background scene.

All the other pixels are considered to be unknown, and then they can be classified into the foreground or background by Markov random field (MRF) optimization.

AUTOMATIC INITIAL SEGMENTATION :AUTOMATIC INITIAL SEGMENTATION : Trimap Labeling Trimap Labeling

Trimap consisting of labels drawn from Trimap labeling method = β-labeling and ξ-labeling β : based on the color difference. zero-mean normalized cross correlation (ZNCC) ξ : based on the foreground model.

ξ : xi . β: beta. μ:mu.

1.If the color of a pixel varies, the pixel should be the background and labeled 2.mathematical morphology is applied to filter out the remained noises such that only one region exists

3.Each pixel whose color differs widely from the background model can be labeled .

Where is a strict threshold to ensure that only the pixels that differ widely from the background model are labeled ξ

The β and μ regions are collected and clustered by using K-means.Let denote the mean color of the th cluster for imageEach pixel p with the label μ in the image is examined and labeled ξ

AUTOMATIC INITIAL SEGMENTATION :AUTOMATIC INITIAL SEGMENTATION : Trimap Labeling Trimap Labeling

β Labeling

ξ Labeling

Background: blackForeground : white

Unknow : gray

Mathematical morphology in (a)

MVI

SEGMENTATION WITH SHAPE SEGMENTATION WITH SHAPE PRPRIORS :IORS :

Volumetric Graph CutsVolumetric Graph Cuts

In (6), the first integral tends toward a photo-consistent surface, while the second, called the ballooning term, prefers a fatter reconstructed model.

For each voxel , let be the photo-consistency score of , where a lower value represents a better photo-consistency. Let be the volume between and the base surface. The true surface is determined by finding the global minimum of the energy function among all candidate surfaces .

Discrete Medial Axis Discrete Medial Axis Constraint:Constraint:

Energy Function Energy Function Analysis:Analysis:Let be a neighborhood system defined for , which containing the

set of all pairs of neighboring voxels.

Let be a family of random variables defined on the set , in which each variable takes a label from

D(p) is the penalty according to how well the voxel fits into the given label, while B(p,q) indicates whether the surface is likely to pass through the edge between p and q .

B(p,q) can maintain the smoothness prior

Discrete Medial Axis Constraint:Discrete Medial Axis Constraint:

Imposing theImposing the DMADMA ConstraintConstraintThe medial axis is represented by a set of discrete voxels interior

to the 3-D object, called discrete medial axis (DMA).

To compute the DMA of the base surface, which is assumed to be an adequate approximation of the DMA of the true surface.

let be the set of voxels in the DMA. Let dp be the minimum distance from the voxel to its nearest voxel in

(12) guarantees that the voxels in are always labeled as being inside the surface.

Discrete Medial Axis Constraint:Discrete Medial Axis Constraint:

Segmentation RefinementSegmentation Refinement

C1.C2.C4 built hull.

C3.C5 projection

EXPERIMENTS:EXPERIMENTS:Initial Segmentation Initial Segmentation

ResultsResultsError

MVI

Trimap Labeling

(β Labeling + ξ Labeling)

Error

EXPERIMENTS:EXPERIMENTS:Learning Shape PriorLearning Shape Prior

Without DMA andballooning increased

a. Visual hull

b. DMA of ( a )

c. MVI

d. Our method

a.Visual hull base surface

b. DMA of ( a )

c.d. reconstructed model

EXPERIMENTS:EXPERIMENTS:Rectification of Segmentation Rectification of Segmentation

ErrorsErrors

Projection of the reconstructed

EXPERIMENTS:EXPERIMENTS:Rectification of Segmentation Rectification of Segmentation

ErrorsErrors

Projection of the reconstructed

Trimap Labeling = β Labeling +ξ Labeling

Automatic initial segmentation

MVI

Shape priors

EXPERIMENTS:EXPERIMENTS:Rectification of Segmentation Rectification of Segmentation

ErrorsErrors

Shape prior1 use 10 images

Shape prior2 use 20 images

Thank you for your Thank you for your

listening !listening !

2008.12.092008.12.09