Optic Flow and Motion Detection

Cmput 615Cmput 615

Martin JagersandMartin Jagersand

Image motion

• Somehow quantify the Somehow quantify the frame-to-frame differences frame-to-frame differences in image sequences.in image sequences.

1.1. Image intensity difference.Image intensity difference.

2.2. Optic flowOptic flow

3.3. 3-6 dim image motion 3-6 dim image motion computationcomputation

Motion is used to:

•Attention:Attention: Detect and Detect and direct using eye and direct using eye and head motions head motions

•Control:Control: Locomotion, Locomotion, manipulation, toolsmanipulation, tools

•Vision:Vision: Segment, depth, Segment, depth, trajectorytrajectory

Small camera re-orientation

Note: Almost all pixels change!

MOVING CAMERAS ARE LIKE STEREO

Locations ofpoints on the object(the “structure”)

The change in spatial locationbetween the two cameras (the “motion”)

Classes of motion

•Still camera, single moving objectStill camera, single moving object

•Still camera, several moving objectsStill camera, several moving objects

•Moving camera, still backgroundMoving camera, still background

•Moving camera, moving objectsMoving camera, moving objects

The optic flow field

•Vector field over the image:Vector field over the image:

[u,v] = f(x,y), u,v[u,v] = f(x,y), u,v = Vel vector, = Vel vector, x,y = x,y = Im posIm pos

•FOE, FOCFOE, FOC Focus of Expansion, Contraction Focus of Expansion, Contraction

Motion/Optic flow vectorsHow to compute?

Im(x + î x; y + î y; t + î t)– Solve pixel correspondence problemSolve pixel correspondence problem

– given a pixel in Im1, look for same pixels in Im2given a pixel in Im1, look for same pixels in Im2

• Possible assumptionsPossible assumptions1.1. color constancycolor constancy: : a point in H looks the same in Ia point in H looks the same in I

– For grayscale images, this is For grayscale images, this is brightness constancybrightness constancy

2.2. small motionsmall motion: points do not move very far: points do not move very far– This is called the This is called the optical flowoptical flow problem problem

Im(x; y; t)

Optic/image flow

Assume: Assume: 1.1. Image intensities from object points remain constant over Image intensities from object points remain constant over

time time

2.2. Image displacement/motion smallImage displacement/motion smallIm(x + î x; y + î y; t + î t) = Im(x; y; t)

Taylor expansion of intensity variation

Keep linear termsKeep linear terms• Use constancy assumption and rewrite:Use constancy assumption and rewrite:

• Notice: Linear constraint, but no unique solutionNotice: Linear constraint, but no unique solution

à@t

@Im = @x@Im; @y

@Imð ñ

á î xî y

ò ó

0 = @x@Imî x + @y

@Imî y + @t@Imî t

Im(x + î x;y + î y; t + î t) = Im(x;y; t) + @x@Imî x + @y

@Imî y + @t@Imî t + h:o:t:

Im(x + î x;y + î y; t + î t) = Im(x;y; t) + @x@Imî x + @y

@Imî y + @t@Imî t + h:o:t:

Aperture problem

• Rewrite as dot productRewrite as dot product

• Each pixel gives one equation in two unknowns:Each pixel gives one equation in two unknowns:n*f n*f = k= k

• Min length solution: Can only detect vectors normal Min length solution: Can only detect vectors normal to gradient directionto gradient direction

• The motion of a line cannot be recovered using only The motion of a line cannot be recovered using only local informationlocal information

à @t@Im

= @x@Im ;@y

@Imð ñ á îxîy

ò ó= r Imá îx

îy

ò ó

fn

f

ît

Aperture problem 2

The flow continuity constraint

• Flows of nearby pixels or Flows of nearby pixels or patches are (nearly) equalpatches are (nearly) equal

• Two equations, two Two equations, two unknowns:unknowns:nn11 * f * f = k= k11

nn22 * f * f = k= k22

• Unique solution Unique solution f f exists, exists, provided provided nn11 and and nn22 not not

parallelparallel

fn

f

f

n

f

Sensitivity to error

• nn11 and and nn22 might be might be almostalmost

parallelparallel• Tiny errors in estimates of Tiny errors in estimates of kk’s ’s

or or nn’s can lead to huge errors ’s can lead to huge errors in the estimate of in the estimate of ff

fn

f

fn

f

Using several points

• Typically solve for motion in 2x2, 4x4 or larger image Typically solve for motion in 2x2, 4x4 or larger image patches. patches.

• Over determined equation system:Over determined equation system:

dIm = MudIm = Mu• Can be solved in least squares sense using Matlab Can be solved in least squares sense using Matlab u = M\dIm u = M\dIm • Can also be solved be solved using normal equations Can also be solved be solved using normal equations u = (Mu = (MTTM)M)-1-1*M*MTTdIm dIm

...à

@t@Im

...

0

@

1

A =

......

@x@Im

@y@Im

......

0

@

1

A î xî y

ò ó

3-6D Optic flow

•Generalize to many freedooms (DOFs)Generalize to many freedooms (DOFs)

All 6 freedoms

X Y Rotation Scale Aspect Shear

M(u) = @Im=@u

M(u) = @Im=@u

Conditions for solvability

– SSD Optimal (u, v) satisfies Optic Flow equationSSD Optimal (u, v) satisfies Optic Flow equation

When is this solvable?• ATA should be invertible • ATA entries should not be too small (noise)• ATA should be well-conditioned• Study eigenvalues:

– 1/ 2 should not be too large (1 = larger eigenvalue)

Edge

– gradients very large or very small– large1, small 2

Low texture region

– gradients have small magnitude

– small1, small 2

High textured region

– gradients are different, large magnitudes

– large1, large 2

Observation

•This is a two image problem BUTThis is a two image problem BUT– Can measure sensitivity by just looking at one of the images!Can measure sensitivity by just looking at one of the images!– This tells us which pixels are easy to track, which are hardThis tells us which pixels are easy to track, which are hard

– very useful later on when we do feature tracking...very useful later on when we do feature tracking...

Errors in Optic flow computation

•What are the potential causes of errors in this procedure?What are the potential causes of errors in this procedure?– Suppose ASuppose ATTA is easily invertibleA is easily invertible– Suppose there is not much noise in the imageSuppose there is not much noise in the image

•When our assumptions are violatedWhen our assumptions are violated– Brightness constancy is Brightness constancy is notnot satisfied satisfied– The motion is The motion is notnot small small– A point does A point does notnot move like its neighbors move like its neighbors

– window size is too largewindow size is too large– what is the ideal window size?what is the ideal window size?

Iterative Refinement

• Used in SSD/Lucas-Kanade tracking algorithmUsed in SSD/Lucas-Kanade tracking algorithm1.1. Estimate velocity at each pixel by solving Lucas-Kanade equationsEstimate velocity at each pixel by solving Lucas-Kanade equations

2.2. Warp H towards I using the estimated flow fieldWarp H towards I using the estimated flow field

- use image warping techniques- use image warping techniques

3.3. Repeat until convergenceRepeat until convergence

Revisiting the small motion assumption

• Is this motion small enough?Is this motion small enough?– Probably not—it’s much larger than one pixel (2Probably not—it’s much larger than one pixel (2ndnd order terms dominate) order terms dominate)– How might we solve this problem?How might we solve this problem?

Reduce the resolution!

image Iimage H

Gaussian pyramid of image H Gaussian pyramid of image I

image Iimage H u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Coarse-to-fine optical flow estimation

image Iimage J

Gaussian pyramid of image H Gaussian pyramid of image I

image Iimage H

Coarse-to-fine optical flow estimation

run iterative L-K

run iterative L-K

warp & upsample

.

.

.

Application: mpeg compression

HW accelerated computation of flow vectors

•Norbert’s trick: Use an mpeg-card to speed Norbert’s trick: Use an mpeg-card to speed up motion computationup motion computation

Other applications:

•Recursive depth recovery: Kostas and JaneRecursive depth recovery: Kostas and Jane

•Motion control (we will cover)Motion control (we will cover)

•SegmentationSegmentation

•TrackingTracking

Lab:

•Assignment1:Assignment1:

•Purpose:Purpose:– Intro to image capture and Intro to image capture and

processingprocessing– Hands on optic flow Hands on optic flow

experienceexperience

•See www page for See www page for details.details.

•Suggestions welcome!Suggestions welcome!

Organizing Optic Flow

Cmput 615Cmput 615

Martin JagersandMartin Jagersand

Questions to think about

Readings: Book chapter, Fleet et al. paper.Readings: Book chapter, Fleet et al. paper.

Compare the methods in the paper and lectureCompare the methods in the paper and lecture

1.1. Any major differences?Any major differences?

2.2. How dense flow can be estimated (how many How dense flow can be estimated (how many flow vectore/area unit)?flow vectore/area unit)?

3.3. How dense in time do we need to sample?How dense in time do we need to sample?

Organizing different kinds of motion

Two examples:Two examples:

1.1. Greg Hager paper: Planar motionGreg Hager paper: Planar motion

2.2. Mike Black, et al: Attempt to find a low Mike Black, et al: Attempt to find a low dimensional subspace for complex motiondimensional subspace for complex motion

Remember:The optic flow field

•Vector field over the image:Vector field over the image:

[u,v] = f(x,y), u,v[u,v] = f(x,y), u,v = Vel vector, = Vel vector, x,y = x,y = Im posIm pos

•FOE, FOCFOE, FOC Focus of Expansion, Contraction Focus of Expansion, Contraction

(Parenthesis)Euclidean world motion -> image

Let us assume there is one rigid object moving withvelocity T and w = d R / dt

For a given point P on the object, we have p = f P/z

The apparent velocity of the point is V = -T – w x P

Therefore, we have v = dp/dt = f (z V – Vz P)/z2

Component wise:

f

ywxywxwfw

z

fTxTfv

f

xwxywywfw

z

fTxTfv

xyzx

yzy

yxzy

xzx

2

2

Motion due to translation:depends on depth

Motion due to rotation:independent of depth

Remember last lecture:

•Solving for the motion of a patchSolving for the motion of a patch

Over determined equation system:Over determined equation system:

ImImtt = Mu = Mu•Can be solved in e.g. least squares sense using Can be solved in e.g. least squares sense using matlab matlab u = M\Imu = M\Imtt

...à

@t@Im

...

0

@

1

A =

......

@x@Im

@y@Im

......

0

@

1

Aî xî y

!

t t+1

3-6D Optic flow

•Generalize to many freedooms (DOFs)Generalize to many freedooms (DOFs)

Im = Mu

Know what type of motion(Greg Hager, Peter Belhumeur)

u’i = A ui + dE.g. Planar Object => Affine motion model:

It = g(pt, I0)

Mathematical Formulation

• Define a “warped image” Define a “warped image” gg– f(p,x) = x’f(p,x) = x’ (warping function), p warp parameters (warping function), p warp parameters– I(x,t)I(x,t) (image a location x at time t) (image a location x at time t)– g(p,Ig(p,Itt) = (I(f(p,x) = (I(f(p,x11),t), I(f(p,x),t), I(f(p,x22),t), … I(f(p,x),t), … I(f(p,xNN),t))’),t))’

• Define the Jacobian of warping functionDefine the Jacobian of warping function– M(p,t) =M(p,t) =

• ModelModel– II00 = g(p = g(ptt, I, It t )) (image I, variation model (image I, variation model gg, parameters , parameters pp))– I = I = MM(p(ptt, I, Itt) ) pp (local linearization (local linearization MM))

• Compute motion parametersCompute motion parameters p = (p = (MMT T MM))-1-1 MMT T I I where where MM = = MM(p(ptt,I,Itt))

@p@I

h i

Planar 3D motion

• From geometry we know that the correct From geometry we know that the correct plane-to-plane transform is plane-to-plane transform is

1.1. for a perspective camera the for a perspective camera the projective projective homographyhomography

2.2. for a linear camera (orthographic, weak-, para- for a linear camera (orthographic, weak-, para- perspective) the perspective) the affine warpaffine warp

uw

vw

" #

= Wa(p;a) =a3 a4

a5 a6

" #

p +a1

a2

" #

u0

v0

" #

= Wh(xh;h) = 1+h7u+h8v1

h1u h3v h5

h2u h4v h6

" #

Planar Texture Variability 1Affine Variability

•Affine warp functionAffine warp function

•Corresponding image variabilityCorresponding image variability

•Discretized for imagesDiscretized for images

= [B 1. . .B 6][y1; . . .;y6]T = Baya

É I a =P

i=16

@ai

@I wÉ ai = @u@I ;@v

@Iâ ã @a1

@u ááá @a6

@u

@a1

@v ááá @a6

@v

" # É a1...É a6

2

4

3

5

É I a = @u@I ;@v

@Iâ ã 1 0 ãu 0 ãv 00 1 0 ãu 0 ãv

" # y1...y6

2

4

3

5

uw

vw

" #

= Wa(p;a) =a3 a4

a5 a6

" #

p +a1

a2

" #

On The Structure of M

u’i = A ui + dPlanar Object + linear (infinite) camera

-> Affine motion model

X Y Rotation Scale Aspect Shear

M(p) = @g=@p

a3 a4

a5 a6

" #

= sR (Ê )a 00 1

" #1 0h 1

" #

Planar Texture Variability 2Projective Variability

• Homography warpHomography warp

• Projective variability:Projective variability:

• WhereWhere ,,

and and

u0

v0

" #

= Wh(xh;h) = 1+h7u+h8v1

h1u h3v h5

h2u h4v h6

" #

É I h = c1

1@u@I ;@v

@Iâ ã u 0 v 0 1 0 à c1

uc2 à c1

vc2

0 u 0 v 0 1 à c1

uc3 à c1

vc3

2

4

3

5É h1...É h8

2

4

3

5

c1 = 1+ h7u + h8vc3 = h2u + h4v+ h6

c2 = h1u + h3v+ h5

= [B 1. . .B 8][y1; . . .;y8]T = Bhyh

Planar motion under perspective projection

•Perspective plane-plane transforms defined by Perspective plane-plane transforms defined by homographieshomographies

Planar-perspective motion 3

• In practice hard to compute 8 parameter model In practice hard to compute 8 parameter model stably from one image, and impossible to find stably from one image, and impossible to find out-of plane variationout-of plane variation

•Estimate variability basis from several images:Estimate variability basis from several images:

Computed EstimatedComputed Estimated

Another idea Black, Fleet) Organizing flow fields

•Express flow field Express flow field f f in in subspace basis subspace basis mm

•Different “mixing” Different “mixing” coefficients coefficients aa correspond to different correspond to different motionsmotions

Example:Image discontinuities

Mathematical formulation

Let:Let:

Mimimize objective function:Mimimize objective function:

==

WhereWhere

Motion basis

Robust error norm

ExperimentMoving camera

•4x4 pixel 4x4 pixel patchespatches

•Tree in Tree in foreground foreground separates well separates well

Experiment:Characterizing lip motion

•Very non-rigid!Very non-rigid!

Summary

• Three types of visual motion extractionThree types of visual motion extraction1.1. Optic (image) flow: Find x,y – image velocitiesOptic (image) flow: Find x,y – image velocities

2.2. 3-6D motion: Find object pose change in image coordinates 3-6D motion: Find object pose change in image coordinates based more spatial derivatives (top down)based more spatial derivatives (top down)

3.3. Group flow vectors into global motion patterns (bottom up)Group flow vectors into global motion patterns (bottom up)

• Visual motion still not satisfactorily solved Visual motion still not satisfactorily solved problemproblem

Sensing and Perceiving Motion

Cmput 610Cmput 610

Martin Martin JagersandJagersand

How come perceived as motion?

Im = sin(t)U5+cos(t)U6 Im = f1(t)U1+…+f6(t)U6

Counterphase sin grating

•Spatio-temporal patternSpatio-temporal pattern– Time Time t, t, Spatial Spatial x,yx,y

s(x;y;t) = A cos(K xcosÊ + K ysinÊ à Ð)cos(! t)

Counterphase sin grating

• Spatio-temporal patternSpatio-temporal pattern– Time Time t, t, Spatial Spatial x,yx,y

Rewrite as dot product:Rewrite as dot product:

= + = +

s(x;y;t) = A cos(K xcosÊ + K ysinÊ à Ð)cos(! t)

21(cos([a;b] x

y

ô õà ! t) + cos([a;b] x

y

ô õ+ ! t)

Result: Standing wave is superposition of two moving waves

Analysis:

•Only one term: Motion left or rightOnly one term: Motion left or right

•Mixture of both: Standing waveMixture of both: Standing wave

•Direction can flip between left and rightDirection can flip between left and right

Reichardt detector

• QT movieQT movie

Severalmotion models

• Gradient: in Gradient: in Computer VisionComputer Vision

• Correlation: In bio Correlation: In bio visionvision

• Spatiotemporal Spatiotemporal filters: Unifying filters: Unifying modelmodel

Spatial response:Gabor function

•Definition:Definition:

Temporal response:

Adelson, Bergen ’85Adelson, Bergen ’85

Note: Terms from Note: Terms from

taylor of taylor of sin(t)sin(t)

Spatio-temporal Spatio-temporal D=DD=DssDDtt

Receptor response toCounterphase grating

•Separable convolutionSeparable convolution

Simplified:

•For our grating: For our grating: (Theta=0)(Theta=0)

•Write as sum of components:Write as sum of components:

= exp(…)*(acos… + bsin…)= exp(…)*(acos… + bsin…)

L s = 2A exp 2

à û2(kà K )2ð ñ

cos(þ à Ð)

Space-time receptive field

Combined cells

• Spat: Temp:Spat: Temp:

• Both:Both:

• Comb:Comb:

Result:

•More directionally specific responseMore directionally specific response

Energy model:

•Sum odd and even phase Sum odd and even phase componentscomponents

•Quadrature rectifierQuadrature rectifier

Adaption:Motion aftereffect

Where is motion processed?

Higher effects:

Equivalence:Reich and Spat

Conclusion

•Evolutionary motion detection is importantEvolutionary motion detection is important

•Early processing modeled by Reichardt detector Early processing modeled by Reichardt detector or spatio-temporal filters.or spatio-temporal filters.

•Higher processing poorly understoodHigher processing poorly understood