Samuel W. Hasinoff Sing Bing Kang Richard Szeliski

Interactive Visual Media GroupMicrosoft Research{sbkang,szeliski}@microsoft.com

Dept. of Computer ScienceUniversity of Torontohasinoff@cs.toronto.edu

Boundary Matting for View Synthesis

2nd Workshop on Image and Video Registration, July 2, 2004

MotivationSuperior view synthesis & 3D editing from N-view stereo

Key approach: occlusion boundaries as 3D curves

• More suitable for view synthesis• Boundaries estimated to sub-pixel

Two major limitations – even with perfect stereo!• Resampling blur• Boundary artifacts

B2B3

Matting problem: Unmix the foreground & background

Matting from Stereo

BFC )1(

Triangulation matting (Smith & Blinn, 1996)

• multiple backgrounds• fixed viewpoint & object

F

B1

Extension to stereo• Lambertian assumption

F

B3B1 B2

underdetermined

Occlusion Boundaries in 3D Model boundaries as 3D splines (currently linear) Assumptions

boundaries are relatively sharp relatively large-scale objects no internal transparency

view 1 view 3view 2 (reference)

3D world

Geometric View of Alpha

alpha partial pixel coverage on F side

simulate blurring by convolving with 2D Gaussian

otherwise,0

0)(,1)(

xdx

),0()(),( Gxx

j

j x)(

alpha depends only on projected 3D curve, x

integration over each pixel

F B

pixel j

Related Work Natural image matting [Chuang et al., 2001]

based on color statistics

Intelligent scissors [Mortenson, 2000]

geometric view of alpha

- single image- user-assisted

Related Work Bayesian Layer estimation [Wexler and Fitzgibbon, 2002]

matting from multiple images using triangulation + priors

- requires very high-quality stereo- alpha calculated at pixel level, only for reference - not suitable for view synthesis

Boundary Matting Algorithm

3D world

view 1 view 3view 2 (reference)

find occlusion boundary in reference view backproject to 3D using stereo depth project to other views initial guess for Bi and F optimize matting

optimize

Initial Boundaries From Stereo Find depth discontinuities Greedily segment longest four-connected curves

Spline control points evenly spaced along curve

Tweak - snap to strongest nearby edge

Background Estimation

F

B1 B2

Use stereo to grab corresponding background-depth pixels from nearby views (if possible)

Color consistency check to avoid mixed pixels

B3

occluded

Foreground Estimation

Invert matting equation, given 3D curve and B

Aggregate F estimates over all views

viewsviews

ii

iii FF

1

2

1

2 )(ˆˆ

BCF )1()(ˆ

BFC )1(

Optimization

Objective: Minimize inconsistency with matting

over curve parameters, x, and foreground colors, F

Pixels with unknown B not included Non-linear least squares, using forward differencing

for Jacobian

views pixels

i j

jijijjiji BαFαCO1 1

2))(1()(),( xxFx

Additional Penalty Terms Favor control points at strong edges

define potential field around each edgel

Discourage large motions (>2 pixels) helps avoid degenerate curves

Naïve object insertion (no matting)

Object insertion with Boundary Matting

Naïve object insertion (no matting)

Object insertion with Boundary Matting

Naïve object insertion (no matting)

Object insertion with Boundary Matting

boundaries calculated with subpixel accuracy

Samsung commercial sequence

Naïve object insertion (no matting)

Object insertion with Boundary Matting

Boundary Matting Naïve method

Boundary Matting Naïve method

boundary mattingboundary matting (sigma = 13)boundary matting (sigma = 26)compositebackgroundno matting

Synthetic Noise

Concluding Remarks

Boundary Matting better view synthesis refines stereo at occlusion boundaries subpixel boundary estimation

Future work incorporate color statistics extend to dynamic setting