creating images the 2-d way jean-françois lalonde april 20, 2010
TRANSCRIPT
Creating images the 2-D way
Jean-François LalondeApril 20, 2010
Creating images (3-D)
Creating images (2-D + 3-D)
Inserting objects into images
Inserting objects into images(2-D + 3-D)
[Debevec, ’98]
Inserting objects in images
Realistic renderings
Expensive and impractical
Highly detailed geometryHighly detailed materials
Very expensive
[Debevec, ’98]
Alternative: Clip art
Easy, intuitive, cheap
Not realistic
Creating images (2-D)
Photo-realistic Cartoon
Expensive and impractical Cheap and intuitive
Image-based rendering
Clip ArtPhoto Clip Art
??
“Photoshop-ing”
Composite by David Dewey
Inserting objects into images
Challenges
object orientationobject orientation
scene illuminationscene illumination
Insert THIS object: impossible!
The use of dataInsert SOME object: much easier!
QuickTime™ and aMPEG-4 Video decompressor
are needed to see this picture.
The Google modelDatabaseDatabase
Sort the objectsSort the objects
QueryQuery ResultsResults
2-D image vs 3-D scene
Outline
Phase I: Database annotation
Phase II: Object insertion
Name: personSubgroup: person, standingHeight: 1.5m Local context: in shadowIllumination: sunny, bright day, no cloudSegmentation quality: excellent, >40 pointsUpsampling blur: low
Data source: LabelMe
Online (http://labelme.csail.mit.edu), user-contributed
170,000 objects in 40,000 images
Polygons and names
[Russell et al., 2005]
Data organization
Top-level categories (chosen manually, 16 total)
Second-level categories (from annotations or clustering)
Annotating the objectsAnnotating the objects
Camera parameters
Assumeflat ground plane
all objects on ground
camera roll is negligible (consider pitch only)
Camera parameters: height and orientation
Human height Human height distributiondistribution
1.7 +/- 0.085 m1.7 +/- 0.085 m(National Center for Health (National Center for Health
Statistics)Statistics)
Car height Car height distributiondistribution
1.5 +/- 0.19 m1.5 +/- 0.19 m(automatically (automatically
learned)learned)
Camera parameters
Object heightsDatabase image
Pixel heights Real heights
Object Estimated average height (m)
Car 1.51
Man 1.80
Woman 1.67
Parking meter 1.36
Fire hydrant 0.87
Estimated object heights
1.0 m
Car
0.5 m
Man Woman Parkingmeter
Firehydrant
1.5 m
Geometry is not enough
Illumination context
Database image
Environment map rough approximation
Exact environment map is impossibleApproximations [Khan et al., ‘06]
Illumination contextDatabase image P(pixel|class) CIE L*a*b* histograms
Automatic Photo PopupAutomatic Photo PopupHoiem Hoiem et al.et al., SIGGRAPH ‘05, SIGGRAPH ‘05
Automatic Photo PopupAutomatic Photo PopupHoiem Hoiem et al.et al., SIGGRAPH ‘05, SIGGRAPH ‘05
Illumination nearest-neighbors
Other criteria: local context
Other criteria: segmentation
LabelMe contributors not always reliable
Segmentation quality38 points / polygon 4 points / polygon
Other criteria: blur
Resolution: avoid up-samplingx3 up-sampling
Recap
Phase I: Database annotation
Phase II: Object insertion
3-D height Segmentation Blur
Object properties (used for sorting the database)
Label Illumination contextCluster Local contextCamera
Let’s insert an object!Poor user-provided segmentations
Noticeable seams
SeamsInput Destination image
Result
Visible
seam!
[Perez et al., 2003]
[Perez et al., 2003]
Poisson blending: ideaInput Destination
Result
Enforce boundary color
(seamless result)
Enforce same gradient than input
Why gradients? 1-D example
Regular blending
bright
dark
1-D example
Blending derivatives
Original signals Derivatives
Reintegration results
1-D exampleGradient domainIntensity domain
?
2-D: not so easy
+1
+2
-3
4
0 2
5
-2
Non integrable: sum over a loop Non integrable: sum over a loop ≠ 0≠ 0
Actually happens all the time in Actually happens all the time in practicepractice
2-D: some notation
Finite differences
2-D: a (possible) solution
Least-squares solution:
??
Solution: Poisson equation
Popular because:Solution is obtained by solving a linear system of equations
Can be solved (somewhat) efficiently
‘\’ in matlab
FFT
Multi-grid solvers (approximate, but really fast!)
2-D: a (popular) solution
Results & limitations
Image editing
Some limitationsImages need to be very well aligned
Differences in background “bleed through”
Images from [Perez et al., 2003]
Poisson blending: improvements
Drag-and-Drop Pasting
[Jia et al., 2006]
User-selectedboundary
Poisson blending
Refinedboundary
Poisson blending
Images from [Jia et al., 2006]
Color bleeding
Still not right!
Not so sensitive to shadow direction [Cavanagh, 2005]
Shadow transfer
++++ ====
Database image Shadow estimate Refined shadow
Object alone Shadow alone Object with shadow
User interface
QuickTime™ and aMPEG-4 Video decompressor
are needed to see this picture.
Street accident
Bridge
Painting
Alley
Failure cases
Shadow transferShadow transfer
Porous objectsPorous objects
Failure cases
Best matching objectsBest matching objects
Pros & cons
Pros Cons
3-DMore control(camera, geometry,
lighting...)Complex!
2-D + 3-D RealisticComplex!
Need access to scene
2-D(photoshop)
RealisticComplex!
Need to sort through
images 2-D
(automatic)Easy, intuitive “Generic” objects