),(),(),(),(),(),( yxvyxIyxuyxIyxIyxJ yx Linearizing (assuming small (u,v)):

Brightness Constancy Equation:

The Brightness Constraint

),(),( ),(),( yxyx vyuxIyxJ

Where: ),(),( yxJyxIIt

-=

0 tyx IvIuI

Each pixel provides 1 equation in 2 unknowns (u,v). Insufficient info.

Another constraint: Global Motion Model Constraint

The 2D/3D Dichotomy

Image motion =

Camera induced motion =

+ Independent motions =

Camera motion+

Scene structure

+Independent motions

2D techniques

3D techniques Singularities in

“2D scenes”

Do not model

“3D scenes”

Requires prior model selection

Global Motion Models2D Models:• Affine• Quadratic• Homography (Planar projective transform)

3D Models:• Rotation, Translation, 1/Depth • Instantaneous camera motion models• Essential/Fundamental Matrix• Plane+Parallax

0)()( 654321 tyx IyaxaaIyaxaaI

Example: Affine Motion

Substituting into the B.C. Equation:

yaxaayxvyaxaayxu

654

321

),(),(

==

Each pixel provides 1 linear constraint in 6 global unknowns

0 tyx IvIuI

(minimum 6 pixels necessary)

2 = tyx IyaxaaIyaxaaIaErr )()()( 654321

Least Square Minimization (over all pixels):

Every pixel contributes Confidence-weighted regression

Example: Affine Motion 2 = tyx IyaxaaIyaxaaIaErr )()()( 654321

Differentiating w.r.t. a1 , …, a6 and equating to zero

22222

22222

222

22222

22222

222

yyyyxyxyx

yyyyxyxyx

yyyyxyxyx

yxyxyxxxx

yxyxyxxxx

yxyxyxxxx

IyxyIyIIIyIxyIIyIxyIIxxIIxyIIIxIxIyIxIIIyIIxIII

IIyIxyIIyIIyxyIyIIxyIIIxIxIxyIIxxIIyIIxIIIyIxII

=

6

5

4

3

2

1

aaaaaa

------

ty

ty

ty

tx

tx

tx

IyIIxIIIIyIIxIII

6 linear equations in 6 unknowns:

image Iimage J

aJwwarp refine

a aΔ+

Pyramid of image J Pyramid of image I

image Iimage J

Coarse-to-Fine Estimation

u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

0 tyx IvIuI ==> small u and v ...

Parameter propagation: )2()2(2),(2)2,2(

)2()2(2),(2)2,2(

654

321

yaxaayxvyxvyaxaayxuyxu

====

)(2)(2

654

321

yaxaayaxaa

==

Quadratic – instantaneous approximation to planar motion

Other 2D Motion Models

287654

82

7321

yqxyqyqxqqv

xyqxqyqxqqu

=

=

yyvxxu

yhxhhyhxhh

pHpHy

yhxhhyhxhh

pHpHx

pHHp

PHHP

PPp

Z

-=-=

==

==

===

',' and

'

'

'''

987

654

3

2

987

321

3

1

33Projective – exact planar motion

(Homography H)

0)( 321 -- pHyIxIIpHIpHI yxtyx 0 tyx IvIuI

Panoramic Mosaic ImageOriginal video clip

Generated Mosaic image

Alignment accuracy (between a pair of frames): error < 0.1 pixel

Original

Outliers

Original

Synthesized

Video Removal

ORIGINAL ENHANCED

Video Enhancement

Direct Methods: Methods for motion and/or shape estimation, which recover the unknown parameters directly from measurable image quantities at each pixel in the image.

Minimization step: Direct methods: Error measure based on dense measurable image quantities(Confidence-weighted regression; Exploits all available information)

Feature-based methods: Error measure based on distances of a sparse set of distinct feature matches.

Image gradients The descriptor(4x4 array of 8-bin histograms)

– Compute gradient orientation histograms of several small windows (128 values for each point)

– Normalize the descriptor to make it invariant to intensity change– To add Scale & Rotation invariance:

Determine local scale (by maximizing DoG in scale and in space), local orientation as the dominant gradient direction.

Example: The SIFT Descriptor

D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. IJCV 2004

• Compute descriptors in each image Find descriptors matches across images

Estimate transformation between the pair of images.• In case of multiple motions:

Use RANSAC (Random Sampling and Consensus) to compute Affine-transformation / Homography / Essential-Matrix / etc.

Benefits of Direct Methods

• High subpixel accuracy.• Simultaneously estimate matches + transformation

Do not need distinct features.• Strong locking property.

Limitations

• Limited search range (up to ~10% of the image size).

• Brightness constancy assumption.

Video Indexing and Editing

The 2D/3D Dichotomy

Image motion =

Camera induced motion =

+ Independent motions =

Camera motion+

Scene structure

+Independent motions

2D techniques

3D techniques Singularities in

“2D scenes”

Do not model

“3D scenes”

A camera-centric coordinate system (R,T,Z)

The Plane+Parallax Decomposition

Original Sequence Plane-Stabilized Sequence

The residual parallax lies on a radial (epipolar) field: wp'

pepipole

p'

Benefits of the P+P Decomposition

• Eliminates effects of rotation

• Eliminates changes in camera parameters / zoom

• Camera parameters: Need to estimate only epipole. (gauge ambiguity: unknown scale of epipole)

• Image displacements: Constrained to lie on radial lines (1-D search problem)

A result of aligning an existing structure in the image.

1. Reduces the search space:

Remove global component which dilutes information !

Translation or pure rotation ???

Benefits of the P+P Decomposition

2. Scene-Centered Representation:

Focus on relevant portion of info

Benefits of the P+P Decomposition

2. Scene-Centered Representation:

Shape = Fluctuations relative to a planar surface in the scene

STAB_RUG SEQ

- fewer bits, progressive encoding

Benefits of the P+P Decomposition

2. Scene-Centered Representation:

Shape = Fluctuations relative to a planar surface in the scene• Height vs. Depth (e.g., obstacle avoidance)

• A compact representation

global (100)component

local [-3..+3]component

total distance [97..103]

camera center scene

• Appropriate units for shape

• Start with 2D estimation (homography).

• 3D info builds on top of 2D info.

3. Stratified 2D-3D Representation:

Avoids a-priori model selection.

Benefits of the P+P Decomposition

Original sequence Plane-aligned sequence Recovered shape

Dense 3D Reconstruction(Plane+Parallax)

Dense 3D Reconstruction(Plane+Parallax)

Original sequence

Plane-aligned sequence

Recovered shape

Original sequence Plane-aligned sequence

Recovered shape

Dense 3D Reconstruction(Plane+Parallax)

Brightness Constancy constraint

P+P Correspondence Estimation

The intersection of the two line constraints uniquely defines the displacement.

1. Eliminating Aperture Problem

Epipolar line

epipole

p

0= TYX IvIuI

other epipolar line

Epipolar line

Multi-Frame vs. 2-Frame Estimation

The two line constraints are parallel ==> do NOT intersect

1. Eliminating Aperture Problem

p

0= TYX IvIuI

anotherepipole

epipole

Brightness Constancy constra

int

The other epipole resolves the ambiguity !