Introduction Tracking with static camera Tracking with moving camera
Tracking
Hakan Ardo
March 4, 2013
Hakan Ardo Tracking March 4, 2013 1 / 57
Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection
Outline
1 IntroductionState spaceSliding Window Detection
2 Tracking with static cameraGreedyKalman filterParticle filter
3 Tracking with moving cameraSelf-motion
Hakan Ardo Tracking March 4, 2013 2 / 57
Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection
State space
State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...
Observation: detection in image x = (x , y , 1)
Observation model: λx = PX
Dynamic model: Xt+1 = Xt + vt and vt+1 = vt
Sthocastic dynamic model: Introduce noise, random numbers q and w
Xt+1 = Xt + vt + q
vt+1 = vt + w
Sthocastic observation model: Introduce noise, a random number r
λx = PX + r
Hakan Ardo Tracking March 4, 2013 3 / 57
Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection
State space
State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...
Observation: detection in image x = (x , y , 1)
Observation model: λx = PX
Dynamic model: Xt+1 = Xt + vt and vt+1 = vt
Sthocastic dynamic model: Introduce noise, random numbers q and w
Xt+1 = Xt + vt + q
vt+1 = vt + w
Sthocastic observation model: Introduce noise, a random number r
λx = PX + r
Hakan Ardo Tracking March 4, 2013 3 / 57
Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection
State space
State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...
Observation: detection in image x = (x , y , 1)
Observation model: λx = PX
Dynamic model: Xt+1 = Xt + vt and vt+1 = vt
Sthocastic dynamic model: Introduce noise, random numbers q and w
Xt+1 = Xt + vt + q
vt+1 = vt + w
Sthocastic observation model: Introduce noise, a random number r
λx = PX + r
Hakan Ardo Tracking March 4, 2013 3 / 57
Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection
State space
State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...
Observation: detection in image x = (x , y , 1)
Observation model: λx = PX
Dynamic model: Xt+1 = Xt + vt and vt+1 = vt
Sthocastic dynamic model: Introduce noise, random numbers q and w
Xt+1 = Xt + vt + q
vt+1 = vt + w
Sthocastic observation model: Introduce noise, a random number r
λx = PX + r
Hakan Ardo Tracking March 4, 2013 3 / 57
Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection
State space
State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...
Observation: detection in image x = (x , y , 1)
Observation model: λx = PX
Dynamic model: Xt+1 = Xt + vt and vt+1 = vt
Sthocastic dynamic model: Introduce noise, random numbers q and w
Xt+1 = Xt + vt + q
vt+1 = vt + w
Sthocastic observation model: Introduce noise, a random number r
λx = PX + r
Hakan Ardo Tracking March 4, 2013 3 / 57
Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection
State space
State: position X = (X ,Y ,Z , 1), velocity, v = (vx , vy , vz , 0), ...
Observation: detection in image x = (x , y , 1)
Observation model: λx = PX
Dynamic model: Xt+1 = Xt + vt and vt+1 = vt
Sthocastic dynamic model: Introduce noise, random numbers q and w
Xt+1 = Xt + vt + q
vt+1 = vt + w
Sthocastic observation model: Introduce noise, a random number r
λx = PX + r
Hakan Ardo Tracking March 4, 2013 3 / 57
Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection
Sliding window detectors
Hakan Ardo Tracking March 4, 2013 4 / 57
Introduction Tracking with static camera Tracking with moving cameraState space Sliding Window Detection
Detection probability
Hakan Ardo Tracking March 4, 2013 5 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Greedy tracker
Hakan Ardo Tracking March 4, 2013 6 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 1
STC Lecture Series
An Introduction to the Kalman FilterGreg Welch and Gary Bishop
University of North Carolina at Chapel Hill
Department of Computer Science
http://www.cs.unc.edu/~welch/kalmanLinks.html
Hakan Ardo Tracking March 4, 2013 7 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Sanjay Patil1 and Ryan Irwin2
Graduate research assistant1,REU undergrad2
Human and Systems Engineering
URL: www.isip.msstate.edu/publications/seminars/msstate/2005/particle/
HUMAN AND SYSTEMS ENGINEERING:Gentle Introduction to Particle Filtering
Hakan Ardo Tracking March 4, 2013 8 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 5
Some Intuition
Hakan Ardo Tracking March 4, 2013 9 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 6
First Estimate
ˆ x 1 = z1
ˆ σ 21 = σ 2z1
Conditional Density Function
14121086420-2
N(z1,σz12 )
z1 σ 2z1
,
Hakan Ardo Tracking March 4, 2013 10 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 7
Second Estimate
Conditional Density Function
z2 σ 2z2
,
ˆ x 2 = ...?
ˆ σ 22 = ...? 14121086420-2
N(z2,σz22 )
Hakan Ardo Tracking March 4, 2013 11 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 8
Combine Estimates
= σ z2
2 σ z1
2 +σ z2
2( )[ ] z1+ σ z1
2 σ z1
2 +σ z2
2( )[ ] z2ˆ x 2= ˆ x 1 + K2 z2 − ˆ x 1[ ]
whereK2 = σ z1
2 σ z1
2 +σ z2
2( )
Hakan Ardo Tracking March 4, 2013 12 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 9
Combine Variances
1 σ 2 = 1 σ z1
2( )+ 1 σ z2
2( )2
Hakan Ardo Tracking March 4, 2013 13 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 10
Combined Estimate Density
ˆ x =
ˆ σ 2 = σ 22
ˆ x 2
14121086420-2
Conditional Density Function
N( σ 2)ˆ x,ˆ
Hakan Ardo Tracking March 4, 2013 14 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 11
Add Dynamics
dx/dt = v + w
wherev is the nominal velocityw is a noise term (uncertainty)
Hakan Ardo Tracking March 4, 2013 15 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Page 7 of 20Particle Filtering – Gentle Introduction and Implementation Demo
Particle filtering algorithm step-by-step (1)
Hakan Ardo Tracking March 4, 2013 16 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Page 8 of 20Particle Filtering – Gentle Introduction and Implementation Demo
Particle filtering step-by-step (2)
Hakan Ardo Tracking March 4, 2013 17 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Page 9 of 20Particle Filtering – Gentle Introduction and Implementation Demo
Particle filtering step-by-step (3)
Hakan Ardo Tracking March 4, 2013 18 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Page 10 of 20Particle Filtering – Gentle Introduction and Implementation Demo
Particle filtering step-by-step (4)
Hakan Ardo Tracking March 4, 2013 19 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Page 11 of 20Particle Filtering – Gentle Introduction and Implementation Demo
Particle filtering step-by-step (5)
Hakan Ardo Tracking March 4, 2013 20 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Page 12 of 20Particle Filtering – Gentle Introduction and Implementation Demo
Particle filtering step-by-step (6)
Hakan Ardo Tracking March 4, 2013 21 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 13
Some Details
xx AAxx wwzz HH xx........
= +=
Hakan Ardo Tracking March 4, 2013 22 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 14
Discrete Kalman Filter
Maintains first two statistical moments
process state (mean)
error covariance
z
y
x
Hakan Ardo Tracking March 4, 2013 23 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 16
Necessary Models
measurementmodel
dynamicmodel
previous state next state
statemeasurement
image plane( u , v )
Hakan Ardo Tracking March 4, 2013 24 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 17
The Process Model
x k+1 = Axk + wk
zk = Hxk + vk
Process Dynamics
Measurement
Hakan Ardo Tracking March 4, 2013 25 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 18
Process Dynamics
x k+1 = Axk + wk
xk ∈ Rn contains the states of the process
state vector
Hakan Ardo Tracking March 4, 2013 26 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 19
Process Dynamics
nxn matrix A relates state at time step k to time step k+1
state transition matrix
x k+1 = Axk + wk
Hakan Ardo Tracking March 4, 2013 27 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 20
Process Dynamics
process noise
wk ∈ Rn models the uncertainty of the process
wk ~ N(0, Q)
x k+1 = Axk + wk
Hakan Ardo Tracking March 4, 2013 28 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 21
Measurement
zk = Hxk + vk
zk ∈ Rm is the process measurement
measurement vector
Hakan Ardo Tracking March 4, 2013 29 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 22
Measurement
zk = Hxk + vk
state vector
Hakan Ardo Tracking March 4, 2013 30 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 23
Measurement
mxn matrix H relates state to measurement
measurement matrix
zk = Hxk + vk
Hakan Ardo Tracking March 4, 2013 31 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 24
Measurement
measurement noise
zk ∈ Rm models the noise in the measurement
vk ~ N(0, R)
zk = Hxk + vk
Hakan Ardo Tracking March 4, 2013 32 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 25
State Estimates
a priori state estimate
a posteriori state estimate
ˆ x –k
ˆ xk
Hakan Ardo Tracking March 4, 2013 33 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 26
Estimate Covariances
a priori estimate error covariance
a posteriori estimate error covariance
Pk– = E[(xk- xk
–)(xk - xk–)T] ˆ ˆ
Pk = E[(xk- xk)(xk - xk)T] ˆ ˆ
Hakan Ardo Tracking March 4, 2013 34 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 27
Filter Operation
Time update (a priori estimates)
Measurement update (a posteriori estimates)
Project state and covariance forwardto next time step, i.e. predict
Update with a (noisy) measurementof the process, i.e. correct
Hakan Ardo Tracking March 4, 2013 35 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 28
Time Update (Predict)
state
error covariance
Hakan Ardo Tracking March 4, 2013 36 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 29
Measurement Update (Correct)
Hakan Ardo Tracking March 4, 2013 37 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 30
Time Update (Predict)
a priori state and error covariance
ˆ xk+1 = Axk–
Pk+1 = APk A + Q–
Hakan Ardo Tracking March 4, 2013 38 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 31
Measurement Update (Correct)
a posteriori state and error covariance
Kalman gain
ˆ xk = xk + Kk (zk - Hxk )ˆ ˆ – –
Pk = (I - Kk H)Pk–
Kk = Pk HT(HPk HT + R)-1 – –
Hakan Ardo Tracking March 4, 2013 39 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
UNC Chapel Hill Computer Science Slide 32
Filter Operation
Time Update(Predict)
Measurement
(Correct)Update
Hakan Ardo Tracking March 4, 2013 40 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Kalman filter tracker
Hakan Ardo Tracking March 4, 2013 41 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Kalman filter tracker
Hakan Ardo Tracking March 4, 2013 42 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Kalman filter tracker
Hakan Ardo Tracking March 4, 2013 43 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Kalman filter tracker
Hakan Ardo Tracking March 4, 2013 44 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Particle filter
Hakan Ardo Tracking March 4, 2013 45 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Patricle filter states
Current state {x
(1)k , x
(2)k , x
(3)k , · · · , x (n)
k
}Predicted state (a priori state extimate){
x−(1)k+1 , x
−(2)k+1 , x
−(3)k+1 , · · · , x
−(n)k+1
}Corrected state (a posteriori state extimate){
x(1)k+1, x
(2)k+1, x
(3)k+1, · · · , x
(n)k+1
}
Hakan Ardo Tracking March 4, 2013 46 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Patricle filter states
Current state {x
(1)k , x
(2)k , x
(3)k , · · · , x (n)
k
}Predicted state (a priori state extimate){
x−(1)k+1 , x
−(2)k+1 , x
−(3)k+1 , · · · , x
−(n)k+1
}Corrected state (a posteriori state extimate){
x(1)k+1, x
(2)k+1, x
(3)k+1, · · · , x
(n)k+1
}
Hakan Ardo Tracking March 4, 2013 46 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Patricle filter states
Current state {x
(1)k , x
(2)k , x
(3)k , · · · , x (n)
k
}Predicted state (a priori state extimate){
x−(1)k+1 , x
−(2)k+1 , x
−(3)k+1 , · · · , x
−(n)k+1
}Corrected state (a posteriori state extimate){
x(1)k+1, x
(2)k+1, x
(3)k+1, · · · , x
(n)k+1
}
Hakan Ardo Tracking March 4, 2013 46 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Dynamic model
Any sthocastic model from which we can sample
f (xk+1 |xk )
Example: The dynamic model from the kalman filter
xk+1 = Axk + wk
f (xk+1 |xk ) = N(Axk ,Q)
Hakan Ardo Tracking March 4, 2013 47 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Dynamic model
Any sthocastic model from which we can sample
f (xk+1 |xk )
Example: The dynamic model from the kalman filter
xk+1 = Axk + wk
f (xk+1 |xk ) = N(Axk ,Q)
Hakan Ardo Tracking March 4, 2013 47 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Particle filter prediction step
Propagate each particle i , separately
The prediction x−(i)k+1 is choosen as a sample from
f(xk+1
∣∣∣x (i)k
)
Hakan Ardo Tracking March 4, 2013 48 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Measurement model
Any sthocastic model from which we can calculate likelihoods
f (zk |xk )
Example: The measurement model from the kalman filter
zk = Hxk + vk
f (zk |xk ) = N(Hxk ,R)
Hakan Ardo Tracking March 4, 2013 49 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Measurement model
Any sthocastic model from which we can calculate likelihoods
f (zk |xk )
Example: The measurement model from the kalman filter
zk = Hxk + vk
f (zk |xk ) = N(Hxk ,R)
Hakan Ardo Tracking March 4, 2013 49 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Particle filter correction step
The measurement model gives a weight for each particle, i
w(i)k = f
(zk
∣∣∣x−(i)k
)Resample the set of particles using the weights
Each corrected particle, x(i)k , is choosen randomly
x(i)k = x
−(j)k for some random j
The probability of choosing sample j is
w(j)k∑
l w(l)k
The same particle may be choosen several times
Hakan Ardo Tracking March 4, 2013 50 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Particle filter correction step
The measurement model gives a weight for each particle, i
w(i)k = f
(zk
∣∣∣x−(i)k
)Resample the set of particles using the weights
Each corrected particle, x(i)k , is choosen randomly
x(i)k = x
−(j)k for some random j
The probability of choosing sample j is
w(j)k∑
l w(l)k
The same particle may be choosen several times
Hakan Ardo Tracking March 4, 2013 50 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Particle filter correction step
The measurement model gives a weight for each particle, i
w(i)k = f
(zk
∣∣∣x−(i)k
)Resample the set of particles using the weights
Each corrected particle, x(i)k , is choosen randomly
x(i)k = x
−(j)k for some random j
The probability of choosing sample j is
w(j)k∑
l w(l)k
The same particle may be choosen several times
Hakan Ardo Tracking March 4, 2013 50 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Particle filter correction step
The measurement model gives a weight for each particle, i
w(i)k = f
(zk
∣∣∣x−(i)k
)Resample the set of particles using the weights
Each corrected particle, x(i)k , is choosen randomly
x(i)k = x
−(j)k for some random j
The probability of choosing sample j is
w(j)k∑
l w(l)k
The same particle may be choosen several times
Hakan Ardo Tracking March 4, 2013 50 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Particle filter correction step
The measurement model gives a weight for each particle, i
w(i)k = f
(zk
∣∣∣x−(i)k
)Resample the set of particles using the weights
Each corrected particle, x(i)k , is choosen randomly
x(i)k = x
−(j)k for some random j
The probability of choosing sample j is
w(j)k∑
l w(l)k
The same particle may be choosen several times
Hakan Ardo Tracking March 4, 2013 50 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Particle filter
Hakan Ardo Tracking March 4, 2013 51 / 57
Introduction Tracking with static camera Tracking with moving cameraGreedy Kalman filter Particle filter
Particle filter tracker
Hakan Ardo Tracking March 4, 2013 52 / 57
Introduction Tracking with static camera Tracking with moving cameraSelf-motion
Recording sport events
Hakan Ardo Tracking March 4, 2013 53 / 57
Introduction Tracking with static camera Tracking with moving cameraSelf-motion
Self-motion
Detect keypoints
Track each keypoint separately
Use RANSAC to find camera motion
Compensate for camera motion (stiching)
Hakan Ardo Tracking March 4, 2013 54 / 57
Introduction Tracking with static camera Tracking with moving cameraSelf-motion
Self-motion
Hakan Ardo Tracking March 4, 2013 55 / 57
Introduction Tracking with static camera Tracking with moving cameraSelf-motion
Self-motion
Hakan Ardo Tracking March 4, 2013 56 / 57
Introduction Tracking with static camera Tracking with moving cameraSelf-motion
Master Thesis: Automatisk inspelning av sport
Detta projekt syftar till att utveckla mjukvara som automatiskt kan filma,folja och mata tider for ett idrottsevenemang, till exempel ett 100m-lopp ifriidrott. For detta kravs dels kunskaper i matematisk bildanalys, men endel kunskaper i programmering kommer aven att behovas. Vi far hjalp avIFK Lund att spela in film och utvardera resultatet.
Ett examensarbete syftar till att utveckla algoritmer for automatiskfoljning av lopp. Fran borjan ar kameran stilla och efter starten skallden hela tiden folja loparna genom att rotera och zooma. Algortimenmaste kunna kompensera och den egna kamerarorelsen.Ytterligare ett examensarbete behandlar automatisk detektion avloparbanan. Genom kamerakalibrering och banans kanda matt kanloparnas hastighet beraknas och diagram over hur hastighetenforandrats genom loppet presenteras for publik och tranare. Dennainformation efterfragas av aktiva lopare. Automatisk detektion avmallinje ar aven viktigt for tidtagningen ovan.
Contact: Petter Strandmark <[email protected]>Hakan Ardo Tracking March 4, 2013 57 / 57