a highly deformable object model for tracking
DESCRIPTION
A Highly Deformable Object Model for Tracking. Nilanjan Ray Department of Computing Science University of Alberta. Overview of Presentation. Tracking deformable objects Motivations: desirable properties of a deformable object model An example application (mouse heart tracking) - PowerPoint PPT PresentationTRANSCRIPT
A Highly Deformable Object Model for Tracking
Nilanjan Ray
Department of Computing Science
University of Alberta
Overview of Presentation
• Tracking deformable objects– Motivations: desirable properties of a deformable object model– An example application (mouse heart tracking)
• Some technical background– Level set function and its application in image processing– Non-parametric probability density function (pdf) estimation– Similarity/dissimilarity measures for pdfs
• Proposed deformable object model and tracking technique
• Results, comparisons and demos
• Ongoing investigations– Incorporating color cues, and other features– Adding constraints on object shape– Application in morphing (?)
• Summary
• Acknowledgements
Tracking Deformable Objects
• Motivations for a new deformable model– desirable properties:– Adapt with deformations (sometimes drastic
deformations, depending on applications)– Ability to learn object and background:
• Ability to separate foreground and background• Ability to recognize object from one image frame to the next,
in an image sequence
Motivations: Some Existing Deformable Models
• Deformable models: – Highly deformable
• Examples: snake or active contour, B-spline snakes, …• Good deformation, but poor recognition (learning) ability
– Not-so-deformable• Examples
– Active shape and appearance models
– G-snake
– …
• Good recognition (learning) capability, but of course poor deformation ability
So, how about good deformation and good recognition capabilities?
Technical Background: Level Set Function
• A level set function represents a contour or a front geometrically• Consider a single-valued function (x, y) over the image domain;
intersection of the x-y plane and represents a contour:
(X(x, y), Y(x, y)) is the point on the curve that is closest to the (x, y) point
• Matlab demo (lev_demo.m)
otherwise,,)),(()),((
objecttheinsideis),(if,)),(()),((),(
22
22
yxYyyxXx
yxyxYyyxXxyx
Applications of Level Set: Image Segmentation
• Matlab segmentation demo (yezzi.m)
• Vessel segmentation
• Brain reconstruction
• Virtual endoscopy
• Trachea fly through
• …tons out there
Show videos
Level Set Applications: Computer Graphics
• Morphing• Simulation• Animation• ….
http://www.sci.utah.edu/stories/2004/fall_levelset.html
Go to http://graphics.stanford.edu/~fedkiw/for amazing videos
More Applications of Level Set Methods
• Go to http://math.berkeley.edu/~sethian/2006/Applications/Menu_Expanded_Applications.html
Technical Background: Non-Parametric Density Estimation
dxdyiyxI
CiH
i
)2
)),((exp(
1)(
2
2
I(x, y) is the image intensity at (x, y)i is the standard deviation of the Gaussian kernelC is a normalization factor that forces H(i) to integrate to unity
Normalized image intensity histogram:
Technical Background: Similarity and Dissimilarity Measures for PDFs
dzzP
zQzQQPKL )
)(
)(log()(),(
dzzPzQQPBC )()(),(
Kullback-Leibler (KL) divergence (a dissimilarity measure):
Bhattacharya coefficient (a similarity measure):
P(z) and Q(z) are two PDFs being compared
Proposed Method: Deformable Object Model and Tracking
• Deformable Object model:
– From the first frame learn the joint pdf of level set function and image intensity (image feature)
• Tracking:
– From second frame onward search for similar joint pdf
Proposed Deformable Object Model
• Joint probability density estimation with Gaussian kernels:
dxdyiyxJlyx
CilQ
il
)2
)),((exp()
2
)),((exp(
1),(
2
2
2
2
J(x, y) is the image intensity at (x, y) point on the first image frame(x, y) is the value of level set function at (x, y) on the first image frameC is a normalization factor
We learn Q on the first video frame given the object contour (representedby the level set function)
Level set function value: lImage intensity: i
Proposed Object Tracking
• On the second (or subsequent) frame compute the density:
• Match the densities P and Q by KL-divergence:
• Minimize KL-divergence by varying the level set function (x, y)
dxdyiyxIlyx
CilP
il
)2
)),((exp()
2
)),((exp(
1),(
2
2
2
2
I(x, y) is the image intensity at (x, y) on the second/subsequent frame(x, y) is the level set function at on the second/subsequent frame
dldiilP
ilQilQKL
),(
),(log),(
Note that here only P is a function of (x, y)
Minimizing KL-divergence
• In order to minimize KL-divergence we use Calculus of variations
• After applying Calculus of variations the rule of update (gradient descent rule) for the level set function becomes:
dldiiyxIlyxlyx
ilP
ilQtyxyx
il
t
l
ttt )
2
)),((exp()
2
)),((exp(
)),((
),(
),()(),(),(
2
2
2
2
21
t : iteration numbert : timestep size
Minimizing KL-divergence: Implementation
• There is a compact way of expressing the update rule:
)),(),,()()((),(),( 11 yxIyxg
P
Qtyxyx ttt
Where g1 is a convolution kernel:
)2
exp()2
exp()(2
2
2
2
21ill
illlg
is a function defined simply as:
),(
),(
il
ilQ
P
Q
P
Q
convolution
dldiiyxIlyxlyx
ilP
ilQtyxyx
il
t
l
ttt )
2
)),((exp()
2
)),((exp(
)),((
),(
),()(),(),(
2
2
2
2
21
Minimizing KL-divergence: A Stable Implementation
• The previous implementation is called explicit scheme and is unstable for large time steps; if small time step is used then the convergence will be extremely slow
• One remedy is a semi-implicit scheme of numerical implementation:
)),(),,()()((1
)),(),,()()((),(),(1
yxIyxgPQ
t
yxIyxgPlQ
tyxyx
t
t
Where g is a convolution kernel:
)2
exp()2
exp()(2
2
2
2
il
illg
In this numerical scheme t can be large and still the solution willbe convergent; So very quick convergence is achieved in this scheme
is a function defined simply as:
),(
),(
il
ilQl
P
lQ
P
lQ
Results: Tracking Cardiac Motion
A few cine MRI frames and delineated boundaries on them
Show videos
Numerical Results and Comparison
10 20 30 40 50 60 70 80 900
0.5
1
Frame number
Pra
tt FO
M
GVF snake method
Proposed method10 20 30 40 50 60 70 80 90
0
0.5
1
Frame number
Seg
men
tatio
n S
core
GVF snake method
Proposed method
20 40 60 80 100 120 140 160 1800
0.5
1
Frame number
Prat
t FOM
GVF snake method
Proposed method 20 40 60 80 100 120 140 160 1800
0.5
1
Seg
men
tatio
n S
core
Frame number
GVF snake method
Proposed method
Pratt’s FOM Segmentation Score
Slow Seq. Rapid Seq. Slow Seq. Rapid Seq.
GVF Snake Method 0.51 0.62 0.51 0.44
Proposed Method 0.88 0.77 0.74 0.79
Comparison of mean performance measures
Sequence with slow heart motion
Sequence with rapid heart motion
Extensions: Tracking Objects in Color Video
• If we want to learn joint distribution of level set function and color channels (say, r, g, b), then non-parametric density estimation suffers from:– Slowness– Curse of dimensionality
• Another important theme is to combine edge information and region information for objects
• One remedy sometimes is to take a linear combination of r, g, and b channels– Fisher’s linear discriminant can be used to learn the
coefficients of linear combination
Extensions: Adding Object Shape Constraint
• Can we constrain the object shape in this computational framework?
),(),()( pqKLPQKLE
dxdylyx
Klq
l
)2
)),((exp(
1)(
2
2
dxdylyx
Klp
l
)2
)),((exp(
1)(
2
2
Frame number: 1
50 100 150 200
20
40
60
80
100
120
140
160
180
Frame number: 2
50 100 150 200
20
40
60
80
100
120
140
160
180
Minimize:
where
Application in Computer Graphics: Morphing
(J1, 1) (I2, 2)
Initial object Shape and intensity/texture
Final object Shape and intensity/texture
Morphing: generate realistic intermediate tuples (It, t)
(I1, 1), (I2, 2), …..
Morphing: Formulation
• Generate intermediate shapes, i.e., level set function t (say, via interpolation):
• Next, generate intermediate intensity It by maximizing:
• Once again we get a similar PDE for It
11
21 ,
1
t
tt
)),(),,((
)),(),,(()(
22
11
tt
ttt
IPDFJPDFBC
IPDFJPDFBCIE
Morphing: Preliminary Results
Source
20 40 60 80 100 120
20
40
60
80
100
120
Target
20 40 60 80 100 120
20
40
60
80
100
120
Show videos
Summary
• Contributions– A novel deformable object model for tracking– Related efficient and stable computation
• Several exciting extensions
• Application in morphing