![Page 1: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/1.jpg)
Data-driven methods: Video & Texture
© A.A. Efros
CS194: Image Manipulation & Computational Photography Alexei Efros, UC Berkeley, Fall 2015
![Page 2: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/2.jpg)
Michel Gondry train video
http://www.youtube.com/watch?v=0S43IwBF0uM
![Page 3: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/3.jpg)
Weather Forecasting for Dummies™ Let’s predict weather:
• Given today’s weather only, we want to know tomorrow’s • Suppose weather can only be {Sunny, Cloudy, Raining}
The “Weather Channel” algorithm: • Over a long period of time, record:
– How often S followed by R – How often S followed by S – Etc.
• Compute percentages for each state: – P(R|S), P(S|S), etc.
• Predict the state with highest probability! • It’s a Markov Chain
![Page 4: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/4.jpg)
4.04.02.03.03.04.01.06.03.0
Markov Chain
What if we know today and yestarday’s weather?
![Page 5: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/5.jpg)
Text Synthesis
[Shannon,’48] proposed a way to generate English-looking text using N-grams: • Assume a generalized Markov model • Use a large text to compute prob. distributions of
each letter given N-1 previous letters • Starting from a seed repeatedly sample this Markov
chain to generate new letters • Also works for whole words
WE NEED TO EAT CAKE
![Page 6: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/6.jpg)
Mark V. Shaney (Bell Labs)
Results (using alt.singles corpus): • “As I've commented before, really relating to
someone involves standing next to impossible.”
• “One morning I shot an elephant in my arms and kissed him.”
• “I spent an interesting evening recently with a grain of salt”
![Page 7: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/7.jpg)
Video Textures
Arno Schödl Richard Szeliski
David Salesin Irfan Essa
Microsoft Research, Georgia Tech
![Page 8: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/8.jpg)
Still photos
![Page 9: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/9.jpg)
Video clips
![Page 10: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/10.jpg)
Video textures
![Page 11: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/11.jpg)
Problem statement
video clip video texture
![Page 12: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/12.jpg)
Our approach
• How do we find good transitions?
![Page 13: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/13.jpg)
Finding good transitions
• Compute L2 distance Di, j between all frames
Similar frames make good transitions
frame i vs.
frame j
![Page 14: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/14.jpg)
Markov chain representation
2 3 41
Similar frames make good transitions
![Page 15: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/15.jpg)
Transition costs
• Transition from i to j if successor of i is similar to j
• Cost function: Ci→j = Di+1, j
• i
j
i+1
j-1
i j→ Di+1, j
![Page 16: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/16.jpg)
Transition probabilities
•Probability for transition Pi→j inversely related to cost:
•Pi→j ~ exp ( – Ci→j / σ2 )
high σ low σ
![Page 17: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/17.jpg)
Preserving dynamics
![Page 18: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/18.jpg)
Preserving dynamics
![Page 19: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/19.jpg)
Preserving dynamics
• Cost for transition i→j
• Ci→j = wk Di+k+1, j+k
Σ
k = -N
N-1
i
j j+1
i+1 i+2
j-1j-2
i j→Di, j-1 D Di+1, j i+2, j+1
i-1
Di-1, j-2
![Page 20: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/20.jpg)
Preserving dynamics – effect
• Cost for transition i→j
• Ci→j = wk Di+k+1, j+k
Σ
k = -N
N-1
![Page 21: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/21.jpg)
2 3 41
Dead ends
• No good transition at the end of sequence
![Page 22: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/22.jpg)
2 3 41
Future cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 23: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/23.jpg)
2 3 41
Future cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 24: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/24.jpg)
2 3 41
Future cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 25: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/25.jpg)
2 3 41
Future cost
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
![Page 26: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/26.jpg)
2 3 41
• Propagate future transition costs backward
• Iteratively compute new cost
• Fi→j = Ci→j + α mink Fj→k
• Q-learning
Future cost
![Page 27: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/27.jpg)
Final result
![Page 28: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/28.jpg)
Finding good loops
• Alternative to random transitions
• Precompute set of loops up front
![Page 29: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/29.jpg)
Video portrait
• c.f. Harry Potter
![Page 30: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/30.jpg)
Region-based analysis
• Divide video up into regions
• Generate a video texture for each region
![Page 31: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/31.jpg)
User selects target frame range
User-controlled video textures
slow variable fast
![Page 32: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/32.jpg)
Video-based animation
• Like sprites computer games
• Extract sprites from real video
• Interactively control desired motion
©1985 Nintendo of America Inc.
![Page 33: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/33.jpg)
Video sprite extraction
![Page 34: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/34.jpg)
Ci j = + angle α βCi j→ →
vector tomouse pointer
Similarity term Control term
velocity vector
Animation{ {
Video sprite control
• Augmented transition cost:
![Page 35: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/35.jpg)
Fi j→
Fi j→
Fi j→ Fi j→
Fi j→
Fi j→Fi j→
SW
W
NWN
NE
E
SES
Goal
Video sprite control
• Need future cost computation
• Precompute future costs for a few angles.
• Switch between precomputed angles according to user input
• [GIT-GVU-00-11]
![Page 36: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/36.jpg)
Interactive fish
![Page 37: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/37.jpg)
Summary / Discussion
• Some things are relatively easy
![Page 38: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/38.jpg)
Discussion
• Some are hard
![Page 39: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/39.jpg)
“Amateur” by Lasse Gjertsen
http://www.youtube.com/watch?v=JzqumbhfxRo similar idea: http://www.youtube.com/watch?v=MsBMG-p1HDM&feature=share&list=PLFFD733D0FF425290
![Page 40: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/40.jpg)
Hyperlapse Videos http://research.microsoft.com/en-
us/um/redmond/projects/hyperlapserealtime/
![Page 41: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/41.jpg)
Texture
• Texture depicts spatially repeating patterns • Many natural phenomena are textures
radishes rocks yogurt
![Page 42: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/42.jpg)
Texture Synthesis • Goal of Texture Synthesis: create new samples of
a given texture • Many applications: virtual environments, hole-
filling, texturing surfaces
![Page 43: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/43.jpg)
The Challenge
• Need to model the whole spectrum: from repeated to stochastic texture
repeated
stochastic
Both?
![Page 44: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/44.jpg)
Efros & Leung Algorithm
• Assuming Markov property, compute P(p|N(p)) – Building explicit probability tables infeasible
p
Synthesizing a pixel
non-parametric sampling
Input image
– Instead, we search the input image for all similar neighborhoods — that’s our pdf for p
– To sample from this pdf, just pick one match at random
![Page 45: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/45.jpg)
Some Details • Growing is in “onion skin” order
– Within each “layer”, pixels with most neighbors are synthesized first
– If no close match can be found, the pixel is not synthesized until the end
• Using Gaussian-weighted SSD is very important – to make sure the new pixel agrees with its closest
neighbors – Approximates reduction to a smaller neighborhood
window if data is too sparse
![Page 46: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/46.jpg)
Neighborhood Window
input
![Page 47: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/47.jpg)
Varying Window Size
Increasing window size
![Page 48: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/48.jpg)
Synthesis Results french canvas rafia weave
![Page 49: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/49.jpg)
More Results white bread brick wall
![Page 50: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/50.jpg)
Homage to Shannon
![Page 51: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/51.jpg)
Hole Filling
![Page 52: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/52.jpg)
Extrapolation
![Page 53: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/53.jpg)
Summary • The Efros & Leung algorithm
– Very simple – Surprisingly good results – Synthesis is easier than analysis! – …but very slow
![Page 54: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/54.jpg)
p
Image Quilting [Efros & Freeman]
• Observation: neighbor pixels are highly correlated
Input image
non-parametric sampling
B
Idea: unit of synthesis = block • Exactly the same but now we want P(B|N(B))
• Much faster: synthesize all pixels in a block at once
• Not the same as multi-scale!
Synthesizing a block
![Page 55: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/55.jpg)
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocks constrained by overlap
B1 B2
Minimal error boundary cut
![Page 56: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/56.jpg)
min. error boundary
Minimal error boundary overlapping blocks vertical boundary
_ = 2
overlap error
![Page 57: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/57.jpg)
Our Philosophy
• The “Corrupt Professor’s Algorithm”: – Plagiarize as much of the source image as you can – Then try to cover up the evidence
• Rationale: – Texture blocks are by definition correct samples of
texture so problem only connecting them together
![Page 58: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/58.jpg)
![Page 59: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/59.jpg)
![Page 60: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/60.jpg)
![Page 61: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/61.jpg)
![Page 62: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/62.jpg)
![Page 63: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/63.jpg)
![Page 64: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/64.jpg)
Failures (Chernobyl Harvest)
![Page 65: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/65.jpg)
input image
Portilla & Simoncelli
Wei & Levoy Our algorithm
Xu, Guo & Shum
![Page 66: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/66.jpg)
Portilla & Simoncelli
Wei & Levoy Our algorithm
Xu, Guo & Shum
input image
![Page 67: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/67.jpg)
Portilla & Simoncelli
Wei & Levoy Our algorithm
input image
Xu, Guo & Shum
![Page 68: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/68.jpg)
Political Texture Synthesis!
![Page 69: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/69.jpg)
Fill Order
• In what order should we fill the pixels?
![Page 70: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/70.jpg)
Fill Order
• In what order should we fill the pixels? – choose pixels that have more neighbors filled – choose pixels that are continuations of lines/curves/edges
Criminisi, Perez, and Toyama. “Object Removal by Exemplar-based Inpainting,” Proc. CVPR, 2003.
![Page 71: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/71.jpg)
Exemplar-based Inpainting demo
http://research.microsoft.com/vision/cambridge/i3l/patchworks.htm
![Page 72: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/72.jpg)
+ =
Application: Texture Transfer
• Try to explain one object with bits and pieces of another object:
![Page 73: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/73.jpg)
Texture Transfer Constraint
Texture sample
![Page 74: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/74.jpg)
• Take the texture from one image and “paint” it onto another object
Texture Transfer
Same as texture synthesis, except an additional constraint: 1. Consistency of texture 2. Similarity to the image being “explained”
![Page 75: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/75.jpg)
= +
![Page 76: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/76.jpg)
![Page 77: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/77.jpg)
Image Analogies
Aaron Hertzmann1,2
Chuck Jacobs2
Nuria Oliver2
Brian Curless3
David Salesin2,3
1New York University 2Microsoft Research 3University of Washington
![Page 78: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/78.jpg)
Image Analogies
A A’
B B’
![Page 79: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/79.jpg)
![Page 80: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/80.jpg)
Blur Filter
![Page 81: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/81.jpg)
Edge Filter
![Page 82: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/82.jpg)
A A’
B B’
Artistic Filters
![Page 83: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/83.jpg)
Colorization
![Page 84: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/84.jpg)
Texture-by-numbers
A A’
B B’
![Page 85: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/85.jpg)
Super-resolution
A A’
![Page 86: Data-driven methods: Video & Texturecs194-26/fa15/Lectures/... · 2015. 10. 20. · Weather Forecasting for Dummies™ Let’s predict weather: • Given today’s weather only, we](https://reader033.vdocument.in/reader033/viewer/2022060905/60a08eaca7bce6695f293341/html5/thumbnails/86.jpg)
Super-resolution (result!)
B B’