multimedia programming 14: matting departments of digital contents sang il park
TRANSCRIPT
![Page 1: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/1.jpg)
Multimedia Programming 14:
MattingDepartments of Digital
ContentsSang Il Park
![Page 2: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/2.jpg)
Midterm Statistics
![Page 3: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/3.jpg)
Outline• Background Subtraction• Matting & compositing
![Page 4: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/4.jpg)
Moving in Time• Moving only in time, while not moving in space, has
many advantages– No need to find correspondences– Can look at how each ray changes over time– In science, always good to change just one variable at a
time
• This approach has always interested artists (e.g. Monet)
• Modern surveillance video camera is a great source of information– There are now many such WebCams now, some running for
several years!
![Page 5: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/5.jpg)
Image Stack
• As can look at video data as a spatio-temporal volume– If camera is stationary, each line through time
corresponds to a single ray in space– We can look at how each ray behaves – What are interesting things to ask?
t0
255time
![Page 6: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/6.jpg)
Example
![Page 7: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/7.jpg)
Getting the right pixels
Average image
Median Image
![Page 8: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/8.jpg)
Webcams
• Lots of cool potential projects – Weather morphing, weather extrapolation, etc.
http://sv.berkeley.edu/view/
![Page 9: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/9.jpg)
Input Video
![Page 10: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/10.jpg)
Average Image
What is happening?
![Page 11: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/11.jpg)
Average/Median Image• What can we do with this?
![Page 12: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/12.jpg)
Background Subtraction
--
==
![Page 13: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/13.jpg)
Crowd Synthesis
1. Do background subtraction in each frame2. Find and record “blobs”3. For synthesis, randomly sample the
blobs, taking care not to overlap them
![Page 14: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/14.jpg)
Background Subtraction• A largely unsolved problem…
Estimatedbackground
Difference Image
ThresholdedForeground on blue
One videoframe
![Page 15: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/15.jpg)
How does Superman fly?
• Super-human powers?• OR• Image Matting and Compositing?
![Page 16: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/16.jpg)
Image Compositing
![Page 17: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/17.jpg)
Compositing Procedure1. Making Sprite (Image + Mask Image (alpha channel))
2. adding the sprite on the background (Iinear Interpolation)
α
![Page 18: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/18.jpg)
Compositing Procedure
1-α α
![Page 19: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/19.jpg)
Multiple Compositing1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)
Composite by David Dewey
2. Blend them into the composite (in the right order)
![Page 20: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/20.jpg)
Compositing: Two Issues
Semi-transparent objects
Pixels too large
![Page 21: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/21.jpg)
Solution: alpha channel• Add one more channel:
– Image(R,G,B,alpha) Sprite!
• Encodes transparency (or pixel coverage):– Alpha = 1: opaque object (complete coverage)– Alpha = 0: transparent object (no coverage)– 0<Alpha<1: semi-transparent (partial
coverage)
• Example: alpha = 0.7
Partial coverage or semi-transparency
![Page 22: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/22.jpg)
Multiple Alpha Blending
• So far we assumed that one image (background) is opaque.
• If blending semi-transparent sprites (the “A over B” operation):
• Icomp = aIa + (1-a)bIb• comp = a + (1-a)b
![Page 23: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/23.jpg)
“Pulling a Matte”• Problem Definition:
– The separation of an image C into• A foreground object image Co,
• a background image Cb,
• and an alpha matte
– Co and can then be used to composite the foreground object into a different image
• Hard problem– Even if alpha is binary, this is hard to do
automatically (background subtraction problem)
– For movies/TV, manual segmentation of each frame is infeasible
– Need to make a simplifying assumption…
![Page 24: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/24.jpg)
Blue Screen
![Page 25: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/25.jpg)
Blue Screen matting• Most common form of matting in TV
studios & movies
• Petros Vlahos invented blue screen matting in the 50s. His Ultimatte® is still the most popular equipment. He won an Oscar for lifetime achievement.
• A form of background subtraction:– Need a known background– Compute alpha as SSD(C,Cb) > threshold
• Or use Vlahos’ formula: = 1-p1(B-p2G)
– Hope that foreground object doesn’t look like background
• no blue ties!
![Page 26: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/26.jpg)
The Ultimatte
p1 and p2
![Page 27: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/27.jpg)
Blue screen for superman?
![Page 28: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/28.jpg)
Topics for the Term Project
![Page 29: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/29.jpg)
Feature based Image Metamorphosis
• Thaddeus Beier and Shawn Neely, “Feature-Based Image Metamorphosis “, SIGGRAPH 1992
![Page 30: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/30.jpg)
Tour into the picture
• Anjyo, K., Horry, Y., and Arai, K. “Tour into the picture”, SIGGRAPH 1997
![Page 31: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/31.jpg)
Texture synthesis
• A. A. Efros and T. K. Leung, "Texture Synthesis by Non-parametric Sampling", ICCV 99• Wei and Levoy, “Fast Texture Synthesis using Tree-structured Vector Quantization “,
SIGGRAPH 2000
![Page 32: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/32.jpg)
Image Analogy
• A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Salesin, “Image Analogy”,SIGGRAPH 2001
![Page 33: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/33.jpg)
Colorization
• Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller, ”Transferring color to grayscale images”, SIGGRAPH 2002
![Page 34: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/34.jpg)
Image Inpainting
• M. Bertalmío, G. Sapiro, V. Caselles and C. Ballester. “Image Inpainting”,SIGGRAPH 2000
![Page 35: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/35.jpg)
Image Blending
• Patrick Perez, Michel Gangnet, Andrew Blake, “Poisson Image Editing”, SIGGRAPH 2003
• http://www.cs.tau.ac.il/~tommer/adv-graphics/ex1.htm
source images Poisson seamless cloning
![Page 36: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/36.jpg)
Painterly Rendering
• Aaron Hertzmann, "Painterly Rendering with Curved Brush Strokes of Multiple Sizes“, SIGGRAPH 1998
![Page 37: Multimedia Programming 14: Matting Departments of Digital Contents Sang Il Park](https://reader035.vdocument.in/reader035/viewer/2022081603/5697c02b1a28abf838cd8d3a/html5/thumbnails/37.jpg)
Painterly Rendering2
• Michio Shiraishi and Yasushi Yamaguchi , “Image Moment-Based Stroke Placement”, ACM SIGGRAPH 99 Conference abstracts and applications, 1999