1

Recent progress in optical flow

Recent progress in optical flow

progres

s

Presented by: Darya Frolova and Denis Simakov

2

Optical Flow is not in favorOptical Flow is not in favor

Often not using Optical Flow is stated as one of the main advantages of a method

Optical Flow methods have a reputation of either unreliable or slow

Very popular slide:

Optical Flow can be computed fast and accurately

Recent works claimRecent works claim:

3

Horn&Schunck

Optical Flow Research: TimelineOptical Flow Research: Timeline

Lucas&Kanade

1981 1992 1998 now

Benchmark:Barron et.al.

Benchmark:Galvin et.al.

many methods

more methods

no significant improvement, but a lot of useful ingredients were developed

Seminal papers

4

In This LectureIn This LectureWe will describe :

Ingredients for an accurate and robust optical flow

How people combine these ingredients

Fast algorithms

• Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”)

A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005• High accuracy optical flow estimation based on a theory for warping T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005• Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 -

2005• Towards ultimate motion estimation: Combining highest accuracy with real-time

performance A. Bruhn, J. Weickert, 2005

• Bilateral filtering-based optical flow estimation with occlusion detection J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006

Papers:

5

What is Optical Flow?What is Optical Flow?

…

6

DefinitionsDefinitions

The optical flowoptical flow is a velocity field in the image which transforms one image into the next image in a sequence [ [ Horn&SchunckHorn&Schunck ] ]

The motion field … is the projection into the image of three-dimensional motion vectors [ [ Horn&SchunckHorn&Schunck ]]

+ =

frame #1 frame #2flow field

7

Ambiguity of optical flowAmbiguity of optical flow

Frame 1Frame 1

flow (1): true motionflow (2)

8

Applications

• video compression• 3D reconstruction• segmentation• object detection• activity detection• key frame extraction• interpolation in time

optical flow

We are usually interested in actual motion

motion field

9

OutlineOutline

Ingredients for an accurate and robust optical flow

Local image constraints on motion

Robust statistics

Spatial coherence

How people combine these ingredients

Fast algorithms

10

Local image constraintsLocal image constraints

11

Brightness ConstancyBrightness Constancy

0),,()1,,( tyxItvyuxI

frame t

frame t+1

)1,,( tvyux

),,( tyx

v

u

12

),,()( tyx

)( )( )( )( )( tyxLIN IvIuI

1 v

uIII tyx

Complex dependence on

vu , Linearize:

Linearized brightness constancyLinearized brightness constancy

),,()1,,( tyxItvyuxI Deviation from brightness constancy (we want to minimize it)

13

Linearized brightness constancyLinearized brightness constancy

2

2

2

tytxt

tyyxy

txyxx

IIIII

IIIII

IIIII

J J – “motion tensor”, or “structure tensor”

1 v

uIII tyxLIN

Let us square the difference:

1 ]1 [

1 1 2 v

uJvuv

uIII

III

vu tyx

t

y

x

LIN

14

Averaged linearized constraintAveraged linearized constraint

1 ]1 [2 v

uJvuLIN

2

2

2

tytxt

tyyxy

txyxx

IIIII

IIIII

IIIII

J

J is a function of x, y (a matrix for every point)

J *= JG

Combine over small neighborhoods (more robust to noise):

15

Method of Lucas&KanadeMethod of Lucas&Kanade

1 ) ( ]1 [2 v

uJGvuGAUSSLIN

Solve independently for each point [ Lucas&Kanade 1981 ]

2min GAUSSLIN linear system

Can be solved for every point where matrix is not degenerate

01

2

2

v

u

IIIII

IIIIIG

tyyxy

txyxx

16

Lukas&Kanade - ResultsLukas&Kanade - Results

flow field flow field

Hamburg taxiRubik cube

17

Brightness is not always constantBrightness is not always constant

Rotating cylinder

Brightness constancy does not always hold

),,()1,,( tyxItvyuxI

inte

nsity

position

),,()1,,( tyxItvyuxI

Gradient constancy holds

inte

nsity

de

rivat

ive

position

18

Local constraints - SummaryLocal constraints - Summary

We have seen

• brightness constancy

0),,()1,,( tyxItvyuxI

linearized

averaged linearized

0),,()1,,( tyxItvyuxI

• gradient constancy

01

) ( ]1 [2

v

uJGvuGAUSSLIN

01

]1 [2

v

uJvuLIN

averaged linearized

01

) ( ]1 [2

v

uJGvuGAUSSLIN

19

Local constraints are not enough!

20

Local constraints work poorlyLocal constraints work poorly

input videoOptical flow direction using

only local constraints

color encodes direction as marked on the boundary

21

Where local constraints failWhere local constraints fail

Uniform regions

Motion is not observable in the image (locally)

22

Where local constraints failWhere local constraints fail

“Aperture problem”We can estimate only one flow component (normal)

23

Where local constraints failWhere local constraints fail

Occlusions

We have not seen where some points moved

Occluded regions are marked in red

24

Obtaining support from neighborsObtaining support from neighbors

Two main problems with local constraints:

• information about motion is missing in some points => need spatial coherency

• constraints do not hold everywhere => need methods to combine them robustly

good

missing

wrong

25

Robust combination of partially reliable data

or: How to hold elections

26

Toy exampleToy example

Find “best” representative for the set of numbers

min2

iixxEL2: min

iixxEL1:

xix

Influence of xi on E: xi → xi + ∆

)mean( ixx Outliers influence the most

ixx proportional to

xxEE ioldnew 2

)median( ixx Majority decides

equal for all xi

oldnew EE

27

Elections and robust statistics Elections and robust statistics

many ordinary people a very rich man

Oligarchy

Votes proportional to the wealth

Democracy

One vote per person

wealth

like in L2 norm minimization like in L1 norm minimization

28

Combination of two flow constraintsCombination of two flow constraints

robust: L1

x

robust in presence of outliers– non-smooth: hard to analyze

usual: L2

easy to analyze and minimize– sensitive to outliers

2x

robust regularized

smooth: easy to analyze robust in presence of outliers

22 x

ε

video

warpedwarped IIII min

),,( ; )1,,( tyxIItvyuxIIwarped

[A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance

29

Spatial PropagationSpatial Propagation

30

Obtaining support from neighborsObtaining support from neighbors

good

missing

wrong

Two main problems with local constraints:

• information about motion is missing in some points => need spatial coherency

• constraints do not hold everywhere => need methods to combine them robustly

31

Homogeneous propagationHomogeneous propagation

video

vu22

min- flow in the x direction- flow in the y direction- gradient

),,( tyxu),,( tyxv

[Horn&Schunck 1981]

This constraint is not correct on motion boundaries => over-smoothing of the resulting flow

32

Robustness to flow discontinuitiesRobustness to flow discontinuities

(also known as isotropic flow-driven regularization)

video

vu )||||(min 22

:

[T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004]High accuracy optical flow estimation based on a theory for warping

22 x

ε

33

Selective flow filteringSelective flow filtering

[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006]Bilateral filtering-based optical flow estimation with occlusion detection

Solution: use “bilateral” filter in space, intensity, flow; taking into account occlusions

We want to propagate information

• without crossing image and flow discontinuities

• from “good” points only (not occluded)

34

Bilateral filterBilateral filter

x

I

x

I

x

I

x

I

Unilateral (usual) Bilateral

Preserves discontinuities!

[C. Tomasi, R. Manduchi, 1998] Bilateral filtering for gray and color images.

35

Using of bilateral filter - ExampleUsing of bilateral filter - Example

occluded regions

cyan rectangle moves to the right and occludes background region marked by red

[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006]Bilateral filtering-based optical flow estimation with occlusion detection

36

Learning of spatial coherenceLearning of spatial coherence

• Come to the next lecture …

37

Spatial coherence: SummarySpatial coherence: Summary

Homogeneous propagation - oversmoothing

Robust statistics with homogeneous propagation - preserves flow discontinuities

Bilateral filtering - combines information from regions with similar flow and similar intensitiesHandles occlusions

38

Two more useful ingredientsTwo more useful ingredients

in brief – one slide each

2D vs. 3D2D vs. 3D

2 frames:

Several frames:

Several frames allow more accurate optical flow estimation

40

MultiscaleMultiscale Optical FlowOptical Flow

(other names: “warping”, “coarse-to-fine”, “multiresolution”)

pyramid for frame 1

tyx IvIuItyxItvyuxI ),,()1,,(

Linearization: valid only for small flow

+

+frame 1warped

?

pyramid for frame 2

upsample

41

MethodsMethods

How to make tasty soup with these ingredients: several recipes

42

OutlineOutline

Ingredients for an accurate and robust optical flow

How people combine these ingredients

Lukas & Kanade meet Horn & Schunck

The more ingredients – the better

Bilateral filtering and occlusions

Fast algorithms

43

Combining ingredientsCombining ingredients

Energy = ∫∫ϕϕ ((Data) + ∫∫ϕϕ ((“Smoothness”)

Local constraints–Brightness constancy

–Image gradient constancy

Spatial coherency–Homogeneous

–Flow-driven

–Bilateral filtering + occlusions

Combined using robust statistics Computed coarse-to-fine

Use several frames

44

Combining Local and GlobalCombining Local and Global

video

vu

JGvu1

) ( ]1 [

videovideo

vuvu

Jvu22

1

]1 [

videovideo

vuvu

Jvu22

1

)(G ]1 [

Lucas&Kanade Horn&Schunk

Basic “Combining local and global” [A. Bruhn, J. Weickert, C. Schnörr, 2002 ]

2

2

2

tytxt

tyyxy

txyxx

IIIII

IIIII

IIIII

JRemember:

45

Sensitivity to noise – quantitative resultsSensitivity to noise – quantitative results

ground truth flow

frame t+1

frame t

Error measure: angle between true and computed flow in

(x,y,t) space

46

The more ingredients - the betterThe more ingredients - the better

videovideo

warpedwarped IIII flow

brightness constancy

gradient constancy

spatial coherence

[Bruhn, Weickert, 2005]Towards ultimate motion estimation: Combining highest accuracy with real-time performance

47

Quantitative resultsQuantitative results

AverageStandard Deviation

Lucas&Kanade 4.3(density 35%)

Horn&Schunk 9.8 16.2

Combining local and global 4.2 7.7

“Towards ultimate …” 2.4 6.7

Angular error

Method

Yosemite sequence with clouds

Average error decreases, but standard deviation is still high….

48

Influence of each ingredientInfluence of each ingredient

For Yosemite sequence with clouds

49

Handling occlusionsHandling occlusions

videovideo

warped smoothnessII

bilateral filteringbilateral filtering of flow:preserve intensity and flow

discontinuities; model occlusions

:

[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, ECCV 2006] Bilateral filtering-based optical flow estimation with occlusion detection

50

Qualitative resultsQualitative results

51

Quantitative resultsQuantitative results

AverageStandard Deviation

Lucas&Kanade 4.3(density 35%)

Horn&Schunk 9.8 16.2

Combining local and global 4.2 7.7

“Towards ultimate …” 2.4 6.7

Bilateral+occlusionsBilateral+occlusions 2.62.6 6.16.1

Angular error

Method

Yosemite sequence with clouds

52

OutlineOutline

Ingredients for an accurate and robust optical flow

How people combine these ingredients

Fast algorithms

Energy functional => discrete equation

Multigrid solver: nearly real-time

53

How to minimize energyHow to minimize energy

dxuuxFuE )',,()(minimize

Necessary condition:

0'

uF

dxd

uF

Necessary condition

Analogy:

Euler-Lagrange equation

)(minimize xf

0)(' xf

54

An exampleAn example

Horn&Schunk

videovideo

vuvu

Jvu22

1

]1 [minimize

Let us see how to derive discretized equation for 1D Horn & Schuhck

image

ttxx uIuIIuIuE2222 ')2()(1D version

(simplified):

55

Iterative minimization Iterative minimization (simple example)(simple example)

0''2 uIIuI txxEuler-Lagrange

Discretized:

Local iterations: )( oldnewi ufu

Linear system of equation for u

02

211

h

uuubua iii

iii

2

2

2

2

211

ha

hb-

ha

uuu

i

i

i

oldi

oldinew

i

image

ttxx uIuIIuIuE2222 ')2()(

56

Life is not a picnicLife is not a picnic

Linear discretized system

image

utxItuxIuE )'( ),()1,( )( Even more complicated

Non-linear in u, non-linear

discretized system

image

ttxx uIuIIuIuE2222 ')2()(

0''2 uIIuI txx

image

ttxx uIuIIuIuE ')2()( 222

0))'('(2 uIIuI dxd

txx

57

Optimization algorithmsOptimization algorithms

Simple iterative minimization

Multigrid: much faster convergence

58

Solving the systemSolving the system

fAu

How to solve?

2 components of success: fast convergence

good initial guess

Start with some initial guess

and apply some iterative method

initialu

59

Relaxation schemes have smoothingsmoothing property:

Only neighboring pixels are coupled in relaxation scheme

It may take thousands of iterations to

propagate information to large distance

. . . . . . . . . . . .

Relaxation smoothes the errorRelaxation smoothes the error

60

Relaxation smoothes the error Relaxation smoothes the error ExamplesExamples

2D case:

1D case:

Error of initial guess Error after 5 relaxation Error after 15 relaxations

61

Idea: coarser gridIdea: coarser grid

On a coarser grid low frequencies become higher

Hence, relaxations can be more effective

initial grid – fine grid

coarse grid – we take every second point

62

Multigrid 2-Level V-CycleMultigrid 2-Level V-Cycle

1. Iterate ⇒ error becomes smooth

2. Transfer error equation to the coarse level ⇒ low frequencies become high

3. Solve for the error on the coarse level ⇒ good error estimation

4. Transfer error to the fine level

5. Correct the previous solution6. Iterate ⇒ remove interpolation artifacts

63

make iteration process faster (on the coarse grid we can effectively minimize the error)

obtain better initial guess (solve directly on the coarsest grid)

Coarse grid - advantagesCoarse grid - advantages

initialu

fAu

Coarsening allows:

go to the coarsest grid

solve here the equation

to findinitialu

interpolate to the

finer grid

initialu

64

Multigrid approach – Full schemeMultigrid approach – Full scheme

65

Non-linear: Full Approximation SchemeNon-linear: Full Approximation Scheme

⇒ Need to transfer current solution to the coarser level

A(·) non-linear

Equation for error involves current solution u0:

Difference from the linear case:

0

0

0 solutioncurrent

Aufr

uu

u

fuA )(

fuA )(fAu

rA fuA )( 0

Linear Non-linear

fine level:

coarse level:

66

Multigrid: SummaryMultigrid: Summary

• Used to solve linear or non-linear equations

• Method: combine two techniques– Basic iterative solver: quickly removes high

frequencies of the error– Coarsening: makes low frequencies high

• Contribution: fast minimization of loosely coupled equations

fAu fuA )(

67

Fast Optimization: ResultsFast Optimization: Results

Gauss-Seidel 1.15 0.87

Full multigrid 0.016 62.8

Gauss-Seidel 9.52 0.11

FAS-multigrid 0.087 11.5

Gauss-Seidel 34.5 0.03

FAS-multigrid 0.396 2.5

Horn&Schunck

CLG

“Towards ultimate…”

Time [sec] frames/sec

68

Summary of the TalkSummary of the Talk• 25 years of Optical Flow : a lot of useful ingredients were developed:

• local constraints:• brightness constancy• gradient constancy

• smoothing techniques:• homogeneous• flow-driven (preserving discontinuities)• bilateral filters• handling of occlusions

• robust functions • multiscale

• All ingredients are combined an a global Energy Minimization approach

• This difficult global optimization can be done very fast using Multigrid

69

Thank you!

70

Coarse grid - advantages

)4(1 2hO

Another advantages of using coarse grids:

• Relaxations are cheaper (because we have less poins) (1/2 as many points in 1D, 1/4 in 2D, 1/8 in 3D)

• Convergence rate is better ( instead of ))(1 2hO

71

Definition of the errorDefinition of the error

fAu true

After each iteration sweep we have a solution: fAu found

There are two measures of the correctness of the solution :

The error: foundtrue uue

The residual:foundAufr

rAe

) )( ( AvfAefAeAvevAAu

foundu

72

Some Approaches to Optical FlowSome Approaches to Optical Flow

),,(),,(),,(),,()1,,( tyxIvtyxIutyxItyxItvyuxI tyx

Gradient-based techniques

use spatiotemporal partial derivatives to estimate flow at each point

frame 1 frame 2

Patch comparison techniquespatch comparison

+ ++ u v

– shown to perform poorly

good performance– aperture problem

Image-drivenImage-driven

isotropic image-driven

2

2

22

1I

+

v+u

image gradient

Idea: relax smoothing constraint on strong image gradients

74

Anisotropic image-driven

anisotropic image-driven

encourages smooth flow along edges and not across

22

2

2

1

I

IID

vDvuDuT

TT

2 IuT

Intuition: penalize only along-the-edge non-smoothness of flow:

Actually used more complex:

75

Flow-driven regularizationFlow-driven regularization

anisotropic flow-driven

)()( vuvideo

video

ss )()( 21

[Aubert, Deriche, Kornprobst, 1995 ]

[Weickert, Schnorr 2000]

an alternative (one of):

• data terms• smoothness terms• complete functionals

76

Flow-driven regularizationFlow-driven regularization

anisotropic flow-drivenisotropic flow-driven

less penalty for non-smooth flow less penalty across flow edges

• data terms• smoothness terms• complete functionals

77

We need smart smoothingWe need smart smoothing

homogeneous

image-driven isotropic

image-driven anisotropic

flow-driven isotropic

flow-driven anisotropic

method

deg

ree

of s

moo

thn

ess

optimal level ?

• data terms• smoothness terms• complete functionals

78

What we getWhat we get

homogeneous

isotropic image-driven

anisotropic image-driven

isotropic flow-driven

anisotropic flow-driven

• data terms• smoothness terms• complete functionals

79

What we payWhat we pay

hom

og

en

eou

s

an

isotr

op

ic im

ag

e-d

riven

isotr

op

ic fl

ow

-dri

ven

isotr

op

ic im

ag

e-d

riven

an

isotr

op

ic fl

ow

-dri

ven

regularization (smoothness) methods

run

nin

g t

ime

60 sec

40 sec

20 sec

(imag

e si

ze 1

60x1

20)

• data terms• smoothness terms• complete functionals

80

min )( uE

u )(uE

energy functional

Discretized equation

Solve by

linear non-linear

local iterations Multigrid (simple for linear, FAS for non-linear)

equation

is a min for

satisfies Euler-Lagrange equation (ELE)

uhence

Energy minimization: overviewEnergy minimization: overview

81

Gauss-Seidel relaxationsGauss-Seidel relaxations

2

2

2

2

211

ha

hb-

ha

uuu

i

i

i

oldi

oldinew

i

• start from random initial guess

• solve for one point assuming its neighbors are fixed (Gauss-Seidel)

initialu

02

211

h

uuubua iii

iii