motion and perceptual organization motion estimation &...
TRANSCRIPT
1
Motion & TrackingMotion is the only Cue
G. Johansson, “Visual Perception of Biological Motion and a Model For Its
Analysis", Perception and Psychophysics 14, 201-211, 1973.
Motion and perceptual organization
• Even “impoverished” motion data can evoke a
strong percept
Motion Estimation & Tracking
Sequence of images contains information about the scene,We want to estimate motion
2
5
Applications
• video compression
• 3D reconstruction
• segmentation
• object detection
• activity detection
• key frame extraction
• interpolation in time
Tracking
Procerus Technologies
http://www.et.byu.edu/groups/magicc/cmsmadesimple/index.php?page=movies
Video Stabilization Structure From Motion
3
Structure From Motion Motion Field & Optical Flow
Optical center
2D motion field
Projection on the image plane of the 3D velocity of the scene
3D motion field
Image intensity
I1
I2
Motion vector - ?
Motion Field & Optical Flow
• Motion field
– The true 2d projection of the 3d motion
• Optical flow
– The measured 2d flow
What we are able to perceive is just an apparent motion, called
Optical flow
(motion, observable only through intensity variations)
Intensity remains constant –no motion is perceived
No object motion, moving light source produces intensity variations
Optical Flow ≠ Motion Field
4
Optical Flow vs. Motion Field
• Optical flow: the measured flow
• Motion field: the real motion
The Barberpole Illusion
Examples of Motion Fields
(a) Motion field of a pilot looking straight ahead while approaching a fixed
point on a landing strip. (b) Pilot is looking to the right in level flight.
(a) (b)
Global Motion
Examples of Motion Fields
(a) (b)
(c) (d)
(a) Translation perpendicular to a surface. (b) Rotation about axis
perpendicular to image plane. (c) Translation parallel to a surface at a
constant distance. (d) Translation parallel to an obstacle in front of a more distant background.
5
)1( +tI
What is Optical Flow?
Optical Flow1p
2p
3p
4p
}{),( iptI
1vr
2vr
3vr
4vr
}{ ivr
Velocity vectors
Source: Stanford CS223B Computer Vision, Winter 2006, lecture notes
Optical Flow Based Search
…
From Darya Frolova and Denis Simakov
Optical flow-methods
Correlation-based techniques - compare
parts of the first image with parts of the second in terms of the similarity in brightness patterns in order to determine the motion
vectors
Feature-based methods - compute and
analyze Optical Flow at small number of well-defined image features
Gradient-based methods - use
spatiotemporal partial derivatives to estimate flow at each point
Optical Flow - Assumptions
• Constant brightness assumption
The apparent brightness of moving objects remains constant
• Small motion
Points do not move very far
• Spatial coherence
– Neighboring points in the scene typically belong to the same surface and hence typically have similar motions
• Temporal coherence
– The image motion of a surface patch changes gradually over time
6
Estimating Optical Flow
• Constant brightness assumption
( )tyxI ,, ( )dttdyydxxI +++ ,,=
Time = t Time = t+dt
* From Lihi Zelnik-Manor
(x,y) (x+dx,y+dy)
( ) tdt
dIv
dy
dIu
dx
dItyxIttvyuxI ∆+++=∆+++ ),,(,,
)tt,vy,ux(I)t,y,x(I ∆+++=
vIuII0 vxt ++=
Take the Taylor series expansion of I :
using brightness assumption:
Constant Brightness Assumption - 2D Case:
Optical Flow Equation
vIuII0 vxt ++=
x
( )t,xI ( )tt,xI ∆+
u
tI
Optical Flow Equation- Intuition
uII xt =−
The change in value It at a pixel P is dependent on:
The distance moved (u).
x
II x ∆
∆=
x∆I∆
tyx vu III −=+
Optical Flow Equation
tIvuI −=⋅∇ ],[
Only the component of the flow in the gradient direction can be determined
The component of the flow parallel to an edge is unknown
7
Optical Flow Equation
Intuition - The Barber Pole Illusion.
true motionvector
perp.component
parallel.component
The Aperture Problem
• Different motions – classified as similar
source: Ran Eshel
The Aperture Problem
• Similar motions – classified as different
source: Ran Eshel
Frame 1
flow (1): true motionflow (2)
From Darya Frolova and Denis Simakov
Ambiguity of Optical Flow
8
tyx vu III −=+
Optical Flow Equation
Shoot! One equation, two velocity unknowns (u,v) ...
Solving for u,v:
Spatial coherence
• Impose additional constraints
– Assume the pixel’s neighbors have the same (u,v)
( ) ( )( ) ( )
( ) ( )
( )( )
( )
−=
−=+
Nt
t
t
NyNx
yx
yx
tyx
v
u
vu
p
p
p
pp
pp
pp
2
1
22
11
I
I
I
II
II
II
III
MMM
AN×2
x2×1
bN×1
p1
pN
bAx =
( ) bAAAxtt 1−
=
bAAxA tt =
p2
Lukas Kanade Scheme
Equivalent to Solving least squares:
ATA ATb
bAx)AA( TT =
• The summations are over all pixels in the K x K window
• This technique was first proposed by Lukas & Kanade (1981)
x
−=
∑∑
∑∑∑∑
ty
tx
yyx
yxx
v
u
II
II
III
III2
2
Lucas-Kanade Equation
I R IxIy It
22×
AAt
12×
x
12×
bAt
9
When can we solve LK Eq ?Optimal (u, v) satisfies Lucas-Kanade equation
• ATA should be invertible
• The eigenvalues of ATA should not be too small (noise)
• ATA should be well-conditioned:
λ1/ λ2 should not be too large (λ1 = larger eigenvalue)
ATA is invertible when there is no aperture problem
Hessian Matrix
Ix = 0
Iy = 0
M =
M =0 0
0 0Non Invertable
Ix = 0
Iy = kM =
0 0
0 k2Non Invertable
Ix = k
Iy = 0M =
k2 0
0 0Non Invertable
Ix = k1
Iy = k2RM =
k2 0
0 0Non Invertable
k1, k2 correlated(R = rotation)
Ix = k1
Iy = k2M =
k12 0
0 k22
Invertable
k1 * k2 = 0
Classification of AtA
50 100 150 200
20
40
60
80
100
120
140
160
180
200
220
2 4 6 8 10 12
2
4
6
8
10
12
large λ1, small λ2
2 4 6 8 10 12
2
4
6
8
10
12
large λ1, large λ2
2 4 6 8 10 12
2
4
6
8
10
12
small λ1, small λ2
=
∑∑∑∑
2
2
yyx
yxxt
III
IIIΑA
The Hessian Matrix
Optical Flow: Iterative Estimation
Optical flow is first order approximation (first order Taylor).
What if first order is not good approx of the area around a pixel?
( ) tdt
dIv
dy
dIu
dx
dItyxIttvyuxI ∆+++=∆+++ ),,(,,
10
Optical Flow: Iterative Estimation
xx0
Estimate:
u0= 0
u1= u0+ û
estimate
update
û
f2(x)f1(x)
Initial guess:
xx0
u1
u2= u1+ û
f1(x-u1) f2(x)
Estimate:
Initial guess: estimate
update
û
Optical Flow: Iterative Estimation
xx0
f2(x)f1(x-u2)
u2
u3= u2+ ûEstimate:
Initial guess: estimate
update
û
Optical Flow: Iterative Estimation
xx0
f1(x-u3) ≈ f2(x)
Optical Flow: Iterative Estimation
11
( ) ( )∑∈
++=Ayx
tyxvuvu
vuvuSSD,
2
,,min,min III
0=++ tyx vu III
Iterative Refinement
Iterative Lukas-Kanade Algorithm
1. Estimate velocity at each pixel by solving Lucas-Kanade
equations:
2. Warp I1 towards I2 using the estimated flow field
3. Repeat until convergence
Find
Optical Flow: Iterative Estimation
• Some Implementation Issues:
– Warp one image, take derivatives of the other
so you don’t need to re-compute the gradient
after each iteration.
– Often useful to low-pass filter the images
before motion estimation (for better derivative
estimation, and linear approximations to
image intensity)
Optical Flow
http://people.csail.mit.edu/lpk/mars/temizer_2001/Optical_Flow/
Optical Flow: Iterative Estimation
12
Optical Flow Revisiting the small motion assumption
• Is this motion small enough?
– Probably not—it’s much larger than one pixel (2nd order terms dominate)
– How might we solve this problem?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Aliasing in large motions
Temporal aliasing causes ambiguities in optical flow because
images can have many pixels with the same intensity.
I.e., how do we know which ‘correspondence’ is correct?
nearest match is
correct (no aliasing)
nearest match is
incorrect (aliasing)
To overcome aliasing: coarse-to-fine estimation.
actual shift
estimated shift
Reduce the resolution!
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
13
image It-1 image I
Gaussian pyramid of image It-1 Gaussian pyramid of image I
image Iimage It-1u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine optical flow estimation
image Iimage J
Gaussian pyramid of image It-1 Gaussian pyramid of image I
image Iimage It-1
Coarse-to-fine optical flow estimation
run iterative L-K
run iterative L-K
warp & upsample
.
.
.
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Optical Flow Results
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
14
Optical Flow for Affine Transformation
+
=
f
c
y
x
ed
ba
v
u
tyx feydxcbyax III −=++⋅+++⋅ )()(
( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
( )
( )
−=
Nt
t
NyNyNyNxNxNx
yyyxxx
f
e
d
c
b
a
yxyx
yxyx
p
p
pppppp
pppppp 1111111
I
I
IIIIII
IIIIII
MM
Optical Flow for Affine Transformation
161666
222
222
22222
22222
22222
22222
×××
−=
∑∑∑∑∑∑
∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑∑
bAx
AA
II
II
II
II
II
II
IIIIIIIII
IIIIIIIII
IIIIIIIII
IIIIIIIII
IIIIIIIII
IIIIIIIII
tt
ty
tx
ty
tx
ty
tx
yyyyxyxyx
yxyxyxxxx
yyyyxyxyx
yxyxyxxxx
yyyyxyxyx
yxyxyxxxx
x
y
y
x
f
e
d
c
b
a
yxyx
yxyx
xxyxxxyx
yyxyyyxy
yyxyyyxy
xxyxxxyx
Errors in Lukas-Kanade
Suppose ATA is easily invertible
Suppose there is minimal noise in the image.
When our assumptions are violated:
– Brightness constancy is not satisfied
– The motion is not small
– A point does not move like its neighbors
• window size is too large
• what is the ideal window size?
What are the potential causes of errors in the LK approach?
Horn and Schunck (1981)
• Two criteria:
– Small error in optical flow constraint equation, Fh(u,v)
– Optical flow is smooth, Fs(u,v)
• Minimize a combined error functional
Fc(u,v) = Fh(u,v) + λ Fs(u,v)
λ is a weighting parameter
Horn & Schunck Optical Flow - Extension
15
Horn & Schunck Optical Flow - Extension
Fc(u,v) = Fh(u,v) + λ Fs(u,v)
Solve by means of calculus of variation (iteratively).
( ) ( ) smoothness from departure :dydxvvuue2
y2
x
D
2y
2xs +++= ∫∫
( ) equation flow opticalin error :2
∫∫ ++=D
tyxc dydxIvIuIe
( ) ( ) min2
2
2
2
2 →∇+∇++⋅∇∫∫ dydxvuII t λu
Fs(u,v) =
Fh(u,v) =
Smoothness term(regularization term)
Data term
Horn and Schunck
• Assumptions
– brightness constancy
– neighboring velocities are nearly identical
• Properties
+ incorporates global information
+ image first derivatives only
- iterative
- smoothes across motion boundaries
1 iteration 4 iteration
16 iteration 64 iteration
Horn & Schunck - Example
1 frame of sequence
Discontinuities
But smoothing term does not allow to save discontinuities
Discontinuities near edges are lost
Synthetic example(method of Horn and Schunck)
Use edge preserving approaches
16
Reconsidering Smoothness Term
( ) ( ) min2
2
2
2
2 →∇+∇++⋅∇∫∫ dydxvuII t λu
Smoothness term(regularization term)
Data term
Use different regularization in smoothness term
Homogeneous propagation
∫ ∇+∇video
vu22
min- flow in the x direction- flow in the y direction- gradient∇
),,( tyxu),,( tyxv
[Horn&Schunck 1981]
This constraint is not correct on motion boundaries => over-smoothing of the resulting flow
Robustness to flow discontinuities
(also known as isotropic flow-driven regularization)
∫ ∇+∇video
vu )||||(min 22φ
:φ
[T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004]
High accuracy optical flow estimation based on a theory for warping
22 ε+x
ε
64
Brightness is not always constant
Rotating cylinder
Brightness constancy does not always hold
),,()1,,( tyxItvyuxI ≠+++
inte
nsity
position
),,()1,,( tyxItvyuxI ∇=+++∇
Gradient constancy holds
inte
nsity
deri
vative
position
17
http://cbia.fi.muni.cz/user_dirs/xulman/gtgen/gt_application.png
Horn & Schunck
https://www.youtube.com/watch?v=Ox8oI7nzSPw&nohtml5=False
Navigation
http://www.et.byu.edu/groups/magicc/cmsmadesimple/index.php?page=movies
Navigation Car Tracking Using Optical Flow
18
Automatic Plane landing
Using Horizon and Optical Flow
https://www.youtube.com/watch?v=U9iy1B5QG-0
Tracking
Some slides are from Sebastian Thrun, Rick Szeliski, Hendrik Dahlkamp, Wolfram Burgard, Kim Chule Hwon, D. Forsyth, M. Isard, T. Darrell Also From Yuri Rapaport
Examples Tracking applications
• Tracking missiles
• Tracking heads/hands/drumsticks
• Extracting lip motion from video
• Lots of computer vision applications
• Economics
• Navigation
SIGGRAPH 2001
19
Detect Foreground objects –
Background subtraction
Compare the current frame to a reference image and label all the different pixels as motion.
Issues:
Detect Foreground objects –
Background subtraction
Examples Tracking: First Idea!
estimateinitial position
x
y
x
y
state dynamics
x
y
measurement
x
y
Starting point Possible new positions Combine Measurementand possibilities to obtainestimate.
20
Tracking
• Bayes Filtering
• Kalman Filtering
• Particle Filtering
On-line posterior density estimation algorithms that estimate the posterior density of the state-space by directly implementing the Bayesian recursion equations.
Particle Filters Condensation Algorithm
Sequential Monte Carlo (SMC)
HUH???
Partially Observable Chains
z2 z3 z4Measurements z1
state x4state x3state x2state x1
)( 0xp
( )tt zzxp ,,1 LWe have to estimate:
Assumptions
• The system follows a Markovian process
• The observations are independent
( ) ( )111 ,, −− = tttt xxpxxxp L
( ) ( )tttt xzpxxzp =,,1 L
21
System Dynamics
t-1 t t+1 time
zt-1 zt zt+1
xt-1 xt xt+1
Measurements(observed)
States(to be estimated)( )1−tt xxp
( )tt xzp
82
Bayes Filters
System state dynamics (uncertain)
Observation dynamics (measurement error)
We are interested in: posterior probability
Estimating system state from noisy observations
( )tt zzxp ,,1 L
( )1−tt xxp
( )tt xzp
Recursive Bayes Filters
• Sequential update of previous estimate
• Allow on-line processing of data
• Rapid adaptation to changing signals
characteristics
• Consists of two steps:
( ) ( )1:11:11 −−− → tttt zxpzxp
( ) ( )ttttt zxpzzxp :11:1 , →−
– Prediction step:
– Update step:
84
Bayes Formula
evidence
prior likelihood
)(
)()|()(
)()|()()|(),(
⋅==
⇒
==
yP
xPxyPyxP
xPxyPyPyxPyxP
22
85
1:( 1) 1:( 1)( | , ) ( | ) ( | )t t t t t t t tp x z z p z x p x zα− −=
1:( 1) 1 1 1:( 1) 1( | ) ( | ) ( | )t t t t t t tp x z p x x p x z dx− − − − −= ∫
1( | )
( | )
t t
t t
p x x
p z x
−Motion Model
Observation Model
Start from: 0 00 0 0
0
( | )( | ) ( )
( )
p z xp x z p x
p z=
Predict:
Update:
Recursive Bayes Filters Bayes Filters
Prior P(x)
Measurement
evidence
P(z|x)
Posterior
P(x|z)
)()|()|( xpxzpzxp ∝
dxxpxxpxp )()|'()'( ∫=
Update
Predict
Predict the new prior p(x’)
On-line posterior density estimation algorithms that estimate the posterior density of the state-space by directly implementing the Bayesian recursion equations.
Particle Filters Condensation Algorithm
Sequential Monte Carlo (SMC)
Uses Particles (many of them) to represent the distributions.
Want to estimate x position of plane flying across a Fiord
Demo – Particle Filtering
Particle Filter Explained without EquationsAndreas Svensson, Uppsala Univ.
23
Demo – Particle Filtering
Want to estimate x position of plane flying across a Fiord
Particle Filter Explained without EquationsAndreas Svensson, Uppsala Univ.
Demo – Particle Filtering
Zt = elevationt - heightt∈ [0; 1].Xt ∈ [0; 5000].Xt can be estimated from Xt-1.Zt can be calculated from Xt .
Velocity distributes uniformly from 800 to 1000 km/h.Assume have map of elevation.
Zt
Xt
Demo – Particle Filtering
Particle Filter Explained without EquationsAndreas Svensson, Uppsala Univ.https://www.youtube.com/watch?v=aUkBa1zMKv4
Particle Filtering
24
Occlusion Challenges
Resolving Occlusion
• Types of occlusions:
– Self occlusion
– Inter-object occlusion
– Occlusion by the background scene structure
“Detection and Tracking of Occluded People”
Siyu Tang, Mykhaylo Andriluka, Bernt Schiele
Use double person detector
25
“Detection and Tracking of Occluded People”
Siyu Tang, Mykhaylo Andriluka, Bernt Schiele
Shadows
Multiple Cameras
• Challenge : How to integrate all the detections from the different cameras
26
source: Ran Eshel