o ptical f low & l ayered m otion prepared by: lidya tonkonogi mu’in rizik 1

1

OPTICAL FLOW & LAYERED MOTIONPrepared by:

Lidya Tonkonogi

Mu’in Rizik

2

INTRODUCTION – MOTION ESTIMATION

Motion estimation algorithms calculate

the motion vectors that describe the

transformation from one image to

another

3

INTRODUCTION – MOTION ESTIMATION

What if… there

were no motion

estimation

algorithms…

4

INTRODUCTION – MOTION ESTIMATION

Why Do we need Motion Estimation?

Video Compression

Video stabilization:

5

INTRODUCTION – MOTION ESTIMATION

Old school Registration algorithms(Pre

optical flow):

Slow and Costly

Failed in processing images with

rotation and other distortions

6

OPTICAL FLOW

7

DEFINITION

OPTICAL FLOW = apparent motion of brightness patterns

Ideally, the optical flow is the projection of thethree-dimensional velocity vectors on the image

8

PROBLEM DEFINITION: OPTICAL FLOW

How to estimate pixel motion from image H to image I?

• Solve pixel correspondence problem– given a pixel in H, look for nearby pixels of the same

color in I

9

Key assumptions• color constancy: a point in H looks the same in I

– For grayscale images, this is brightness constancy

• small motion: points do not move very far

This is called the optical flow problem

10

OPTICAL FLOW CONSTRAINTS (GRAYSCALE IMAGES)

Let’s look at these constraints more closely• brightness constancy: Q: what’s the equation?

H(x,y)=I(x+u, y+v)

11

• small motion: (u and v are less than 1 pixel)– suppose we take the Taylor series expansion

of I:

12

OPTICAL FLOW EQUATION Combining these two equations

In the limit as u and v go to zero, this becomes exact

13

OPTICAL FLOW EQUATION

Q: how many unknowns and equations per pixel?

2 unknowns, one equation

Aperture Problem

14

WHAT’S IS THE PROBLEM

THE MOTION FROM LEFT TO RIGHT ??!

15

WHAT’S IS THE PROBLEM

WRONG !!!

16

THE APERTURE PROBLEM

Different motions – classified as similar

source: Ran Eshel

17

THE APERTURE PROBLEM

Similar motions – classified as different

source: Ran Eshel

18

THE APERTURE PROBLEM This problem results from the fact that we have

only one equation and two unknowns. This equation : (the gradient)

Define a line in (u,v) space

19

To solve this equation we will calculate the Normal flow

We get at most “Normal Flow” – with one point we can only detect movement perpendicular to the brightness gradient.

Solution is to take a patch of pixels Around the pixel of interest.

THE APERTURE PROBLEM

20

APERTURE PROBLEM AND NORMAL FLOW

21

OPTICAL FLOW TECHNIQUES

Differential techniques: Local method:

Lucas and Kanade Global method:

Horn and Schunck

22

INTRODUCTION – MOTION ESTIMATION

Lucas and Kanade

23

Lucas and Kanade proposed an algorithm to solve the aperture problem.

We back to the function And they proposed another solution to find u and v .

Question : How to get more equations for a pixel? Basic idea: impose additional constraints

one method: pretend the pixel’s neighbors have the same (u,v) in other words If we use a kxk window, that gives us equations per pixel!

LUCAS-KANADE ALGORITHM

2k

24

For example :Assume that we use a 5x5 window, that

gives us 25 equations per pixel!

LUCAS-KANADE - EXAMPLE

25

We try to solve this problem by :

In the previous example :

Minimizing

2A u b

26

LUKAS-KANADE FLOWProb:we have more equations than unknowns

Solution: solve least squares problem• minimum least squares solution given by solution

of:

2A u b

K x 2 2 x 1 K x 1

Minimize :

27

CONDITIONS FOR SOLVABILITY Optimal (u, v) satisfies Lucas-Kanade equation

When is This Solvable?• ATA should be invertible • ATA should not be too small due to noise

– eigenvalues l1 and l2 of ATA should not be too small

ATA is solvable when there is no aperture problem

28

Hessian Matrix

Ix = 0Iy = 0

M =

M =0 0

0 0

Non invertible

Ix = 0Iy = k

M =0 0

0 k2

Non invertible

Ix = kIy = 0

M =k2 0

0 0

Non invertible

Ix = k1

Iy = k2RM =

k2 0

0 0

Non invertible

k1, k2 correlated(R = rotation)

Ix = k1

Iy = k2M =

k12 0

0 k22

invertible

k1 * k2 = 0

29

EXAMPLE :

30

EDGE

– large gradients, all the same– large l1, small l2

31

LOW TEXTURE REGION

– gradients have small magnitude– small l1, small l2

32

HIGH TEXTURED REGION

– gradients are different, large magnitudes– large l1, large l2

33

LUCAS-KANADE

Advantages easy and fast calculation accurate time derivatives

Disadvantages errors on boundaries

34

ERRORS IN LUKAS-KANADE What are the potential causes of errors in this procedure?

Suppose ATA is easily invertibleSuppose there is not much noise in the image

When our assumptions are not valid• Brightness constancy is not satisfied• The motion is not small

35

HORN & SCHUNCK

Berthold Horn

36

Horn and Schunk proposed an algorithm to solve the aperture problem by using a Global method with smoothness constraint .

Minimize a global energy functional with calculus of variations

We back to the function And they proposed another solution to find u and v .

HORN-SCHUNCK ALGORITHM

37

HORN-SCHUNCK ALGORITHM

They assumed that in most of the pictures, the velocity of the objects do not change radically between adjacent pixels !

Here our new equation and we want to find the minimum :

2 2 22x y tE I u I v I u v dxdy

38

HORN-SCHUNCK ALGORITHM2

2

2 2

2 2

( ) 0

( ) 0

where is the Laplace operator

x x y t

y x y t

I I u I v I u

I I u I v I v

x y

2 2 2

2 2 2

( , ) ( , ) ( , )

( )

( )

x x y x t

x y y y t

u x y u x y u x y

I u I I v u I I

I I u I v v I I

39

HORN-SCHUNCK ALGORITHM Iterative scheme

The superscript k denotes the current iteration The solution depends on the neighboring values of

the flow field, it must be repeated once the neighbors have been updated. 

12 2 2

12 2 2

( )

( )

k kx x y tk k

x y

k ky x y tk k

x y

I I u I v Iu u

I I

I I u I v Iv v

I I

40

REVISITING THE SMALL MOTION ASSUMPTION

Is this motion small enough? Probably not—it’s much larger than one pixel (2nd order terms

dominate) How might we solve this problem?

41

ALIASING

Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity.

I.e., how do we know which ‘correspondence’ is correct?

nearest match is correct (no aliasing)

nearest match is incorrect (aliasing)

To overcome aliasing: coarse-to-fine estimation.

actual shift

estimated shift

42

REDUCE THE RESOLUTION!

43image Iimage H

Gaussian pyramid of image H Gaussian pyramid of image I

image Iimage H u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

COARSE-TO-FINE OPTICAL FLOW ESTIMATION

44image Iimage J

Gaussian pyramid of image H Gaussian pyramid of image I

image Iimage H

COARSE-TO-FINE OPTICAL FLOW ESTIMATION

run iterative L-K

run iterative L-K

warp & upsample

.

.

.

45

ITERATIVE REFINEMENT• Iterative Lukas-Kanade Algorithm

1. Estimate velocity at each pixel by solving Lucas-Kanade equations

2. Warp I(t-1) towards I(t) using the estimated flow field- use image warping techniques

3. Repeat until convergence

46

OPTICAL FLOW RESULTS

47

OPTICAL FLOW RESULTS

48

LAYERED MOTION

49

WHAT IS LAYERED MOTION?

A system for representing moving

images with sets of overlapping layers

50

WHY DO WE NEED IT?

Smooth Edges

Reduces blurring – Each layer motion

can be smooth

51

52

WHY DO WE NEED IT?

Smooth Edges

Reduces blurring – Each layer motion

can be smooth

Used for video segmentation

53

VIDEO SEGMENTATION

54

HOW DO WE DO IT?

Edward Adelson and John Wang (1994)

55

ADELSON AND WANG

High Level – divide the image into three

maps:

Intensity Map

Alpha Map (transparency)

Velocity Map

56

ADELSON AND WANG

57

ADELSON AND WANG

The garden scene:

58

ADELSON AND WANG

Divide the frame into non-overlapping patches

59

ADELSON AND WANG

Calculate the velocity with a coarse to

fine optical flow algorithm

60

ADELSON AND WANG

Find the affine motion estimation, why?

We assume that most of the motions in

the world we can describe by affine

transformation: Translation Rotation Zoom Shear

61

ADELSON AND WANG

Affine transformation matrix – reminder:

We have six variables we want to find – a, b, c, d, e, f

62

ADELSON AND WANG

One way to calculate them is by

defining matching points between two

images and calculating the affine

transformation matrix

63

ADELSON AND WANG

Another technique is using the velocity

vector, in the following manner:

Lets define as the velocity vector

And our affine transformation matrix:

64

ADELSON AND WANG

Break the velocity vector to X and Y

directions:

65

ADELSON AND WANG

Define a hypothesis vector for every

patch, that will represent best the

affine transformation in that patch:

Where:

66

ADELSON AND WANG

Define our original coordinates vector

as:

Now, we can define our affine motion

as:

67

ADELSON AND WANG

Calculate the using vector, through the

following equation:

68

69

ADELSON AND WANG

The reliability of the hypothesis can be

measured by it’s residual error (let’s skip

it’s calculation)

Patches with a Hypotheses with a residual

error greater than a specified threshold

are eliminated from the calculation (large

residual error usually indicates that the

patch contains object boundaries in it)

70

ADELSON AND WANG

Patches hypotheses with a residual

error smaller than a defined threshold,

and similar affine transformations we

want to group together

71

ADELSON AND WANG

Divide the patches into clusters using

k-means (remember the segmentation

lecture)

72

ADELSON AND WANG

Now we have regions with similar

affine transformation

We want to verify our affine

transformation vs. the actual

movement

73

ADELSON AND WANG

For this purpose we define:

Where is the region where (x, y) located

And is the affine motion field corresponding to

the region hypotheses

74

ADELSON AND WANG

We want to minimize the distortion at

each location, and so for that we will

take the minimal value of:

Where is the minimal value for the region

75

ADELSON AND WANG

Now lets build our Opacity Map using

* Notice that this will give us only two layers, but

this model can be extended to as many layers as

we wish, by assigning each a layer with a number

76

ADELSON AND WANG

We iterate to minimize the distortion

Conversion is obtained when only a

few pixels are reassigned or if we

reached the max number of iterations.

77

ADELSON AND WANG

The Layers representation:

78

ADELSON AND WANG

In the last phase we warp the layers

together, using a median filter to

overcome slight changes in the

intensity

79

ADELSON AND WANG

Final Result:

80

Good Job Adelson and Wang!But can we do better??

81

LAYERED MOTION OPTIMIZATIONS

Optimizations by Weiss and Adelson

(1996)

Automatically define the optimal

number of layers

Replace Affine motion model with

smooth regularized motion models

82

LAYERED MOTION OPTIMIZATIONS

Optimizations by Baker, Szeliski and

Anandan (1998):

Use homographies instead of affine

motion estimation

83

84

LAYERED MOTION OPTIMIZATIONS

Optimizations by Baker, Szeliski and

Anandan (1998) – continue:

Optimize The layers colors and opacity

by minimizing the discrepancies

between the re-synthesized and

observed motion sequence

85

LAYERED MOTION OPTIMIZATIONS

Baker, Szeliski and

Anandan (1998) – Layers

assignment:

Adelson and Wang–

Layers assignment:

86

DEMO TIME!

87

SUMMARY

Motions Estimation Algorithms

Optical Flow

The Aperture Problem

Lucas and Kanade Algorithm

Horn and Schunck Algorithm

Coarse-to-fine

Layered Motion Estimation (Adelson and Wang)

88

REFERENCES

Books:

Computer Vision: Algorithms and

Applications, Richard Szeliski

Introductory Techniques for 3D Computer

Vision, Emanuele Trucco, Alessandro Verri

89

REFERENCES

Papers: Layered Representation for Motion analysis,

John Y.A. Wang & Edward H. Adelsonhttp://persci.mit.edu/pub_pdfs/wang_tr221.pdf

http://en.wikipedia.org/wiki/Horn%E2%80%93Schunck_method

http://en.wikipedia.org/wiki/Lucas%E2%80%93Kanade_method

http://robots.stanford.edu/cs223b04/CS%20223-B%20L9%20Optical%20Flow.ppt

90

REFERENCES

Youtube: https://

www.youtube.com/v/j3uCV0JYMJ4?version=3&start=137&end=242&autoplay=0&hl=en_US&rel=0

https://www.youtube.com/watch?v=sp77AjBdlEc