creating images the 2-d way jean-françois lalonde april 20, 2010

Creating images the 2-D way

Jean-François LalondeApril 20, 2010

Creating images (3-D)

Creating images (2-D + 3-D)

Inserting objects into images

Inserting objects into images(2-D + 3-D)

[Debevec, ’98]

Inserting objects in images

Realistic renderings

Expensive and impractical

Highly detailed geometryHighly detailed materials

Very expensive

[Debevec, ’98]

Alternative: Clip art

Easy, intuitive, cheap

Not realistic

Creating images (2-D)

Photo-realistic Cartoon

Expensive and impractical Cheap and intuitive

Image-based rendering

Clip ArtPhoto Clip Art

??

“Photoshop-ing”

Composite by David Dewey

Inserting objects into images

Challenges

object orientationobject orientation

scene illuminationscene illumination

Insert THIS object: impossible!

The use of dataInsert SOME object: much easier!

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

The Google modelDatabaseDatabase

Sort the objectsSort the objects

QueryQuery ResultsResults

2-D image vs 3-D scene

Outline

Phase I: Database annotation

Phase II: Object insertion

Name: personSubgroup: person, standingHeight: 1.5m Local context: in shadowIllumination: sunny, bright day, no cloudSegmentation quality: excellent, >40 pointsUpsampling blur: low

Data source: LabelMe

Online (http://labelme.csail.mit.edu), user-contributed

170,000 objects in 40,000 images

Polygons and names

[Russell et al., 2005]

Data organization

Top-level categories (chosen manually, 16 total)

Second-level categories (from annotations or clustering)

Annotating the objectsAnnotating the objects

Camera parameters

Assumeflat ground plane

all objects on ground

camera roll is negligible (consider pitch only)

Camera parameters: height and orientation

Human height Human height distributiondistribution

1.7 +/- 0.085 m1.7 +/- 0.085 m(National Center for Health (National Center for Health

Statistics)Statistics)

Car height Car height distributiondistribution

1.5 +/- 0.19 m1.5 +/- 0.19 m(automatically (automatically

learned)learned)

Camera parameters

Object heightsDatabase image

Pixel heights Real heights

Object Estimated average height (m)

Car 1.51

Man 1.80

Woman 1.67

Parking meter 1.36

Fire hydrant 0.87

Estimated object heights

1.0 m

Car

0.5 m

Man Woman Parkingmeter

Firehydrant

1.5 m

Geometry is not enough

Illumination context

Database image

Environment map rough approximation

Exact environment map is impossibleApproximations [Khan et al., ‘06]

Illumination contextDatabase image P(pixel|class) CIE L*a*b* histograms

Automatic Photo PopupAutomatic Photo PopupHoiem Hoiem et al.et al., SIGGRAPH ‘05, SIGGRAPH ‘05

Automatic Photo PopupAutomatic Photo PopupHoiem Hoiem et al.et al., SIGGRAPH ‘05, SIGGRAPH ‘05

Illumination nearest-neighbors

Other criteria: local context

Other criteria: segmentation

LabelMe contributors not always reliable

Segmentation quality38 points / polygon 4 points / polygon

Other criteria: blur

Resolution: avoid up-samplingx3 up-sampling

Recap

Phase I: Database annotation

Phase II: Object insertion

3-D height Segmentation Blur

Object properties (used for sorting the database)

Label Illumination contextCluster Local contextCamera

Let’s insert an object!Poor user-provided segmentations

Noticeable seams

SeamsInput Destination image

Result

Visible

seam!

[Perez et al., 2003]

[Perez et al., 2003]

Poisson blending: ideaInput Destination

Result

Enforce boundary color

(seamless result)

Enforce same gradient than input

Why gradients? 1-D example

Regular blending

bright

dark

1-D example

Blending derivatives

Original signals Derivatives

Reintegration results

1-D exampleGradient domainIntensity domain

?

2-D: not so easy

+1

+2

-3

4

0 2

5

-2

Non integrable: sum over a loop Non integrable: sum over a loop ≠ 0≠ 0

Actually happens all the time in Actually happens all the time in practicepractice

2-D: some notation

Finite differences

2-D: a (possible) solution

Least-squares solution:

??

Solution: Poisson equation

Popular because:Solution is obtained by solving a linear system of equations

Can be solved (somewhat) efficiently

‘\’ in matlab

FFT

Multi-grid solvers (approximate, but really fast!)

2-D: a (popular) solution

Results & limitations

Image editing

Some limitationsImages need to be very well aligned

Differences in background “bleed through”

Images from [Perez et al., 2003]

Poisson blending: improvements

Drag-and-Drop Pasting

[Jia et al., 2006]

User-selectedboundary

Poisson blending

Refinedboundary

Poisson blending

Images from [Jia et al., 2006]

Color bleeding

Still not right!

Not so sensitive to shadow direction [Cavanagh, 2005]

Shadow transfer

++++ ====

Database image Shadow estimate Refined shadow

Object alone Shadow alone Object with shadow

User interface

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

Street accident

Bridge

Painting

Alley

Failure cases

Shadow transferShadow transfer

Porous objectsPorous objects

Failure cases

Best matching objectsBest matching objects

Pros & cons

Pros Cons

3-DMore control(camera, geometry,

lighting...)Complex!

2-D + 3-D RealisticComplex!

Need access to scene

2-D(photoshop)

RealisticComplex!

Need to sort through

images 2-D

(automatic)Easy, intuitive “Generic” objects

Thanks!jlalonde@cs.cmu.edu