video textures arno schödl richard szeliski david salesin irfan essa microsoft research, georgia...

Video TexturesVideo Textures

Arno SchödlRichard Szeliski

David SalesinIrfan Essa

Microsoft Research, Georgia Tech

Still photosStill photos

Video clipsVideo clips

Video texturesVideo textures

Related workRelated work

Image-based rendering

• Novel still views from still images

• View Morphing [Seitz 96]

• Lightfields / Lumigraph [Levoy 96, Gortler 96]

• Façade [Debevec 96]

• Virtualized Reality [Kanade 97]

• many more…

Image-based rendering (IBR)Image-based rendering (IBR)

• From View Morphing by Seitz and Dyer

Slide added by D.Brogan

Related workRelated work

Texture synthesis

• Infinite extension of 2D pattern

• [Heeger 95]

• [de Bonet 97]

• [Efros 99]

Related workRelated work

Temporal texture synthesis

• Video as a spatio-temporal texture

• Statistical Learning of Multidimensional Textures [Bar-Joseph 99][Wei 00]

Related workRelated work

Video Rewrite

• Assemble video snippets to generate lip motion

• [Bregler 97]

Periodic motion analysis

• Uses similar analysis tools

Related workRelated work

• [Niyogi 94]

• [Seitz 97]

• [Polana 97]

• [Cutler 00]

Problem statementProblem statement

video clip video texture

Our approachOur approach

How do we find good transitions?

Finding good transitions Finding good transitions

Compute L2 distance Di, j between all frames

Similar frames make good transitions

frame ivs.

frame j

Markov chain representationMarkov chain representation

2 3 41

Similar frames make good transitions

Transition costs Transition costs

Transition from i to j if successor of i is similar to j

Cost function: Cij = Di+1, j

i

j

i+ 1

j-1

i j D i+ 1, j

Transition probabilitiesTransition probabilities

Probability for transition Pij inversely related

to cost:Pij ~ exp ( – Cij / 2 )

high low

Preserving dynamicsPreserving dynamics

Preserving dynamics Preserving dynamics

Preserving dynamics Preserving dynamics

Cost for transition ij

Cij = wk Di+k+1, j+kk = -N

N -1

i

j j+ 1

i+ 1 i+ 2

j-1j-2

i jD i, j - 1 D Di+ 1, j i+ 2, j+ 1

i-1

D i-1, j-2

Preserving dynamics – effect Preserving dynamics – effect

Cost for transition ij

Cij = wk Di+k+1, j+kk = -N

N -1

2 3 41

Dead endsDead ends

No good transition at the end of sequence

2 3 41

Future costFuture cost

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

2 3 41

Future costFuture cost

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

2 3 41

Future costFuture cost

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

2 3 41

Future costFuture cost

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

2 3 41

• Propagate future transition costs backward

• Iteratively compute new cost

Fij = Cij + mink Fjk

• Q-learning

Future costFuture cost

Future cost – effectFuture cost – effect

Fixed-length videosFixed-length videos

• Alternative to random transitions

• Precompute set of loops up front

Slide edited by D.Brogan

Fixed-length videosFixed-length videos

• Creating fixed-length videos• Enumerate all sets of transitions with a total

length L (multiloops)• Select the ones that are legal (the loops

overlap so video can jump from one to another)

• Find the best one

• Exponential in number of transitions

Slide added by D.Brogan

Fixed-length videosFixed-length videos

• Create an L x N table

• L = length of longest loop being considered

• N = number of loops being considered

Slide added by D.Brogan

Fixed-length videosFixed-length videos

• Populate the table

• For each (loopn, lengthl) tuple, find best loop (possibly compound) of length l that includes at least one instance of loop n

Slide added by D.Brogan

Fixed-length videosFixed-length videos

• Populate the table with dynamic programming

• Start from row one, find best loops

• Proceed through the rows

• (length5, loopC) is found from (length3, loopC) + (length2, loopD)

Slide added by D.Brogan

Fixed-length videosFixed-length videos

• Each cell requires inspection of at most L-1 compound loops from same column and N-1 entries from other columns

• Total Cost: O(L2N2)

Slide added by D.Brogan

Fixed-length videosFixed-length videos

• Organizing the loops

• You cannot jump from any loop to any other loop… sequence order matters

• Start with loop with largest last frame num.

• Remove it from the pool

• Identify isolated loops

• Add loops that require first loop for access

Slide added by D.Brogan

Visual discontinuitiesVisual discontinuities

• Problem: Visible “Jumps”

Crossfading Crossfading

• Solution: Crossfade from one sequence to the other.

A i-2

A i-2

B j-2

15

…

…

3

1 2

2 1

3

4

4 4

4 4

4+ + +

A i-1

A i-1 / B j-2 A i-1 / B j-2 A i-1 / B j-2

B j-1

A i

B j

A i+1

B j+1

B j+1

Morphing Morphing

• Interpolation task:

A25 B2

5 C15+ +

Morphing Morphing

• Interpolation task:

• Compute correspondencebetween pixels of all frames

• Optical flow

A25 B2

5 C15+ +

Slide edited by D.Brogan

Morphing Morphing

• Interpolation task:

• Compute correspondence between pixels of all frames

• Interpolate pixel position andcolor in morphed frame

• based on [Shum 2000]

A25 B2

5 C15+ +

Results – crossfading/morphingResults – crossfading/morphing

Results – crossfading/morphingResults – crossfading/morphing

Jump Cut Crossfade Morph

CrossfadingCrossfading

Frequent jump & crossfadingFrequent jump & crossfading

Video portraitVideo portrait

Useful for web pages

Combine with IBR techniques

Video portrait – 3DVideo portrait – 3D

Region-based analysisRegion-based analysis

• Divide video up into regions

• Generate a video texture for each region

Automatic region analysisAutomatic region analysis

User selects target frame range

User-controlled video texturesUser-controlled video textures

slow variable fast

Lengthen / shorten video without affecting speed

Time warpingTime warping

shorter original longer

Video-based animationVideo-based animation

• Like spritescomputer games

• Extract spritesfrom real video

• Interactively control desired motion

©1985 Nintendo of America Inc.

Video sprite extractionVideo sprite extraction

blue screen m attingand velocity estim ation

C i j = + angle C i j

vector tom ouse pointer

S im ilarity term Contro l term

velocity vector

Animation{ {

Video sprite controlVideo sprite control

• Augmented transition cost:

F i j

F i j

F i j F i j

F i j

F i jF i j

S W

W

N W

N

N E

E

S E

S

G oal

Video sprite controlVideo sprite control

• Need future cost computation

• Precompute future costs for a few angles.

• Switch between precomputed angles according to user input

• [GIT-GVU-00-11]

Interactive fishInteractive fish

Summary Summary

• Video clips video textures

• define Markov process

• preserve dynamics

• avoid dead-ends

• disguise visual discontinuities

Summary Summary

• Extensions

• regions

• external constraints

• video-based animation

Discussion Discussion

• Some things are relatively easy

Discussion Discussion

• Some are hard

Future workFuture work

• Better analysis of complex motions

• New optimality metrics

• More powerful video-based animation

• Other kinds of video-based rendering

A final exampleA final example