blending and compositing computational photography connelly barnes many slides from james hays,...
TRANSCRIPT
![Page 1: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/1.jpg)
Blending and Compositing
Computational Photography
Connelly Barnes
Many slides from James Hays, Alexei Efros
![Page 2: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/2.jpg)
Blending + Compositing
● Previously:○ Color perception in humans, cameras○ Bayer mosaic○ Color spaces (L*a*b*, RGB, HSV)
![Page 3: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/3.jpg)
Blending + Compositing● Today
david dmartin (Boston College)
![Page 4: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/4.jpg)
Compositing Procedure1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)
Composite by David Dewey
2. Blend them into the composite (in the right order)
![Page 5: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/5.jpg)
Need blending
![Page 6: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/6.jpg)
Alpha Blending / Feathering
01
01
+
=Iblend = aIleft + (1-a)Iright
![Page 7: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/7.jpg)
Setting alpha: simple averaging
Alpha = .5 in overlap region
![Page 8: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/8.jpg)
Setting alpha: center seam
Alpha = logical(dtrans1>dtrans2)
DistanceTransformbwdist(MATLAB)
![Page 9: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/9.jpg)
Setting alpha: blurred seam
Alpha = blurred
Distancetransform
![Page 10: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/10.jpg)
Setting alpha: center weighting
Alpha = dtrans1 / (dtrans1+dtrans2)
Distancetransform
Ghost!
![Page 11: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/11.jpg)
Effect of Window Size
0
1 left
right0
1
![Page 12: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/12.jpg)
Affect of Window Size
0
1
0
1
![Page 13: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/13.jpg)
Good Window Size
0
1
“Optimal” Window: smooth but not ghosted
![Page 14: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/14.jpg)
Band-pass filtering
• Laplacian Pyramid (subband images)• Created from Gaussian pyramid by subtraction
Gaussian Pyramid (low-pass images)
![Page 15: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/15.jpg)
Laplacian Pyramid
• How can we reconstruct (collapse) this pyramid into the original image?
Need this!
Originalimage
![Page 16: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/16.jpg)
Pyramid Blending
0
1
0
1
0
1
Left pyramid Right pyramidblend
![Page 17: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/17.jpg)
Pyramid Blending
![Page 18: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/18.jpg)
laplacianlevel
4
laplacianlevel
2
laplacianlevel
0
left pyramid right pyramid blended pyramid
![Page 19: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/19.jpg)
Laplacian Pyramid: Blending
• General Approach:1. Build Laplacian pyramids LA and LB from
images A and B
2. Build a Gaussian pyramid GR from selected region R
3. Form a combined pyramid LS from LA and LB using nodes of GR as weights:• LS(i,j) = GR(I,j,)*LA(I,j) + (1-GR(I,j))*LB(I,j)
4. Collapse the LS pyramid to get the final blended image
![Page 20: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/20.jpg)
Laplacian Pyramid: Example
• Show ongoing research project
![Page 21: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/21.jpg)
Blending Regions
![Page 22: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/22.jpg)
Horror Photo
david dmartin (Boston College)
![Page 23: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/23.jpg)
Chris Cameron
![Page 24: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/24.jpg)
Simplification: Two-band Blending
• Brown & Lowe, 2003– Only use two bands: high freq. and low freq.– Blends low freq. smoothly– Blend high freq. with no smoothing: use
binary alpha
![Page 25: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/25.jpg)
Don’t blend…cut
• So far we only tried to blend between two images. What about finding an optimal seam?
Moving objects become ghosts
![Page 26: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/26.jpg)
Davis, 1998• Segment the mosaic
– Single source image per segment– Avoid artifacts along boundries
• Dijkstra’s algorithm
![Page 27: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/27.jpg)
min. error boundary
Dynamic programming cuts
overlapping blocks vertical boundary
_ =2
overlap error
![Page 28: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/28.jpg)
Graph cuts• What if we want similar “cut-where-things-
agree” idea, but for closed regions?
– Dynamic programming can’t handle loops
![Page 29: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/29.jpg)
Graph cuts (simple example à la Boykov&Jolly, ICCV’01)
n-links
s
t a cuthard constraint
hard constraint
Minimum cost cut can be computed in polynomial time
(max-flow/min-cut algorithms)
![Page 30: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/30.jpg)
Kwatra et al, 2003
Actually, for this example, dynamic programming will work just as well…
![Page 31: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/31.jpg)
Lazy Snapping
Interactive segmentation using graph cuts
![Page 32: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/32.jpg)
Gradient Domain Image Blending
• In Pyramid Blending, we decomposed our image into 2nd derivatives (Laplacian) and a low-res image
• Lets look at a more direct formulation:– No need for low-res image
• captures everything (up to a constant)
– Idea: • Differentiate• Composite• Reintegrate
![Page 33: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/33.jpg)
Gradient Domain blending (1D)
Twosignals
Regularblending
Blendingderivatives
bright
dark
![Page 34: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/34.jpg)
Gradient Domain Blending (2D)
• Trickier in 2D:– Take partial derivatives dx and dy (the gradient field)– Fiddle around with them (smooth, blend, feather, etc)– Reintegrate
• But now integral(dx) might not equal integral(dy)
– Find the most agreeable solution• Equivalent to solving Poisson equation• Can use FFT, deconvolution, multigrid solvers, etc.
![Page 35: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/35.jpg)
Gradient Domain: Math (1D)
f(x)
f’(x) = [-1 0 1] f⊗
f’(x)
![Page 36: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/36.jpg)
Gradient Domain: Math (1D)
f(x)
f’(x) = [-1 0 1] f⊗
f’(x)
![Page 37: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/37.jpg)
Gradient Domain: Math (1D)
f’(x)
g’(x)
Modify f’ to get target gradients g’
![Page 38: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/38.jpg)
Gradient Domain: Math (1D)
g’(x)
Solve for g that has g’ as its gradients
Plus any boundary constraints on g(write on board)
![Page 39: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/39.jpg)
Gradient Domain: Math (1D)
• Sparse linear system (after differentiating)• Solve with conjugate-gradient
or direct solver• MATLAB A \ b, or Python scipy.sparse.linalg
![Page 40: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/40.jpg)
Gradient Domain: Math (2D)Solve for g that has gx as its x derivative,And gy as its y derivative
Slide from Pravin Bhat
![Page 41: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/41.jpg)
Gradient Domain: Math (2D)
Slide from Pravin Bhat
![Page 42: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/42.jpg)
Gradient Domain: Math (2D)
– Output filtered image – f– Specify desired pixel-differences – (gx, gy)– Specify desired pixel-values – d– Specify constraints weights – (wx, wy, wd)
min wx(fx – gx)2 + wy(fy – gy)2 + wd(f – d)2
f
Energy function (derive on board)
From Pravin Bhat
![Page 43: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/43.jpg)
Gradient Domain: Example
• GradientShop by Pravin Bhat:
GradientShop
![Page 44: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/44.jpg)
Gradient Domain: Example
• Gradient domain painting by Jim McCann:
Real-Time Gradient-Domain Painting
![Page 45: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/45.jpg)
Thinking in Gradient Domain
• James McCann Real-Time Gradient-Domain Painting, SIGGRAPH 2009
![Page 46: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/46.jpg)
Perez et al., 2003
![Page 47: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/47.jpg)
Perez et al, 2003
• Limitations:– Can’t do contrast reversal (gray on black ->
gray on white)– Colored backgrounds “bleed through”– Images need to be very well aligned
editing
![Page 48: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/48.jpg)
Putting it all together
• Compositing images– Have a clever blending function
• Feathering• Center-weighted• Blend different frequencies differently• Gradient based blending
– Choose the right pixels from each image• Dynamic programming – optimal seams• Graph-cuts
• Now, let’s put it all together:– Interactive Digital Photomontage, 2004 (video)
![Page 49: Blending and Compositing Computational Photography Connelly Barnes Many slides from James Hays, Alexei Efros](https://reader036.vdocument.in/reader036/viewer/2022070411/56649ca25503460f9496155c/html5/thumbnails/49.jpg)