),(),(),(),(),(),( yxvyxIyxuyxIyxIyxJ yx Linearizing (assuming small (u,v)):

Brightness Constancy Equation:

The Brightness Constraint

),(),( ),(),( yxyx vyuxIyxJ

Where: ),(),( yxJyxIIt

0 tyx IvIuI

Each pixel provides 1 equation in 2 unknowns (u,v). Insufficient info.

Another constraint: Global Motion Model Constraint

Global Motion Models2D Models:• Affine• Quadratic• Planar projective transform (Homography)

3D Models:• Rotation, Translation, 1/Depth • Instantaneous camera motion models• Plane+Parallax

0)()( 654321 tyx IyaxaaIyaxaaI

Example: Affine Motion

Substituting into the B.C. Equation:

yaxaayxv

yaxaayxu

654

321

),(

),(

Each pixel provides 1 linear constraint in 6 global unknowns

0 tyx IvIuI

(minimum 6 pixels necessary)

2 tyx IyaxaaIyaxaaIaErr )()()( 654321

Least Square Minimization (over all pixels):

image Iimage J

aJwwarp refine

a

aΔ+

Pyramid of image J Pyramid of image I

image Iimage J

Coarse-to-Fine Estimation

u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

0 tyx IvIuI ==> small u and v ...

Quadratic – instantaneous approximation to planar motion

Other 2D Motion Models

287654

82

7321

yqxyqyqxqqv

xyqxqyqxqqu

yyvxxu

yhxhh

yhxhhy

yhxhh

yhxhhx

','

and

'

'

987

654

987

321

Projective – exact planar motion

(Homography)

Panoramic Mosaic ImageOriginal video clip

Generated Mosaic image

Alignment accuracy (between a pair of frames): error < 0.1 pixel

Original

Outliers

Original

Synthesized

Video Removal

ORIGINAL ENHANCED

Video Enhancement

Direct Methods:

Methods for motion and/or shape estimation, which recover the unknown parameters directly from measurable image quantities at each pixel in the image.

Minimization step:

Direct methods:

Error measure based on dense measurable image quantities(Confidence-weighted regression; Exploits all available information)

Feature-based methods:

Error measure based on distances of a sparse set of distinct features.

Benefits of Direct Methods

• Subpixel accuracy.• Does not need distinct features.

•Locking property.

Limitations

• Limited search range (up to 10% of the image size).

• Brightness constancy.

Handling Varying Brightness

Preprocessing:• Mean and contrast normalization.• Laplacian pyramids.

Measurable image quantities:• brightness values• correlation surfaces [Irani-Anandan:iccv98, Mandelbaum-et-

al:iccv99]

• mutual information [Viola-et-al]

Video Indexing and Editing

The 2D/3D Dichotomy

Image motion =

Camera induced motion =

+ Independent motions =

Camera motion

+Scene structure

+Independent motions

2D techniques

3D techniques Singularities in

“2D scenes”

Do not model

“3D scenes”

The Plane+Parallax Decomposition

Original Sequence Plane-Stabilized Sequence

The residual parallax lies on a radial (epipolar) field: wp'

p

epipole

p'

Benefits of the P+P Decomposition

• Eliminates effects of rotation

• Eliminates changes in camera parameters / zoom

• Camera parameters: Need to estimate only epipole. (gauge ambiguity: unknown scale of epipole)

• Image displacements: Constrained to lie on radial lines (1-D search problem)

A result of aligning an existing structure in the image.

1. Reduces the search space:

Remove global component which dilutes information !

Translation or pure rotation ???

Benefits of the P+P Decomposition

2. Scene-Centered Representation:

Focus on relevant portion of info

Benefits of the P+P Decomposition

2. Scene-Centered Representation:

Shape = Fluctuations relative to a planar surface in the scene

STAB_RUG SEQ

- fewer bits, progressive encoding

Benefits of the P+P Decomposition

2. Scene-Centered Representation:

Shape = Fluctuations relative to a planar surface in the scene• Height vs. Depth (e.g., obstacle avoidance)

• A compact representation

global (100)component

local [-3..+3]component

total distance [97..103]

camera center

scene

• Appropriate units for shape

• Start with 2D estimation (homography).

• 3D info builds on top of 2D info.

3. Stratified 2D-3D Representation:

Avoids a-priori model selection.

Benefits of the P+P Decomposition

Original sequence Plane-aligned sequence Recovered shape

Dense 3D Reconstruction(Plane+Parallax)

Dense 3D Reconstruction(Plane+Parallax)

Original sequence

Plane-aligned sequence

Recovered shape

Results

Original sequence Plane-aligned sequence

Recovered shape

Brightness Constancy constraint

Multi-Frame vs. 2-Frame Estimation

The intersection of the two line constraints uniquely defines the displacement.

1. Eliminating Aperture Problem

Epipolar line

epipole

p

0 TYX IvIuI

other epipolar line

Epipolar line

Multi-Frame vs. 2-Frame Estimation

The two line constraints are parallel ==> do NOT intersect

1. Eliminating Aperture Problem

p

0 TYX IvIuI

anotherepipole

epipole

Brightness C

onstancy constra

int

The other epipole resolves the ambiguity !

3D Motion Models

ZxTTxxyyv

ZxTTyxxyu

ZYZYX

ZXZYX

)()1(

)()1(2

2

)(1

)(1

233

133

tytt

xyv

txtt

xxu

w

w

Local Parameter:

ZYXZYX TTT ,,,,,

),( yxZ

Instantaneous camera motion:

Global parameters:

Residual Planar Parallax Motion

Global parameters: 321 ,, ttt

),( yxLocal Parameter: