image analysis active contour models (snakes)
TRANSCRIPT
University of Ioannina - Department of Computer Science and Engineering
Christophoros Nikou
Images taken from:
Computer Vision course by Kristen Grauman, University of Texas at Austin.
Image Analysis
Active contour models (snakes)
2
C. Nikou – Image Analysis (T-14)
a.k.a. active contours, snakes
Given: initial contour (model) near desired object
[Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987]
Figure credit: Yuri Boykov
Deformable contours
3
C. Nikou – Image Analysis (T-14)
Given: initial contour (model) near desired object
Goal: evolve the contour to fit exact object boundary
Main idea: elastic band is
iteratively adjusted so as to
• be near image positions with
high gradients, and
• satisfy shape “preferences” or
contour priors
[Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987]
Deformable contours (cont.)
4
C. Nikou – Image Analysis (T-14)
Image from http://www.healthline.com/blogs/exercise_fitness/uploaded_images/HandBand2-795868.JPG
Deformable contours (cont.)
5
C. Nikou – Image Analysis (T-14)
initial intermediate final
Both are useful for shape fitting but they differ signifficanlty:
Hough
Rigid model shape
Single voting pass can
detect multiple instances
Deformable contours
Prior on shape types, but shape
iteratively adjusted (deforms)
Requires initialization nearby
One optimization “pass” to fit a
single contour
Deformable contours vs Hough
6
C. Nikou – Image Analysis (T-14)
• Some objects have similar basic form but
some variety in the contour shape.
Deformable contours (why?)
7
C. Nikou – Image Analysis (T-14)
• Non-rigid,
deformable
objects can
change their
shape over
time, e.g. lips,
hands…
Deformable contours (why?)
8
C. Nikou – Image Analysis (T-14)
• Non-rigid,
deformable
objects can
change their
shape over
time, e.g. lips,
hands…
Deformable contours (why?)
9
C. Nikou – Image Analysis (T-14)
• Non-rigid, deformable objects can change their shape
over time.
Deformable contours (why?)
10
C. Nikou – Image Analysis (T-14)
• Representation of the contours
• Defining the energy functions
– External
– Internal
• Minimizing the energy function
• Extensions:
– Tracking
– Interactive segmentation
Aspects to consider
11
C. Nikou – Image Analysis (T-14)
• Representation of the contours
• Defining the energy functions
– External
– Internal
• Minimizing the energy function
• Extensions:
– Tracking
– Interactive segmentation
Aspects to consider
12
C. Nikou – Image Analysis (T-14)
• We’ll consider a discrete representation of the contour,
consisting of a list of 2D point positions (“vertices”).
),,( iii yx
1,,1,0 ni for
• At each iteration, we’ll have the
option to move each vertex to
another nearby location (“state”).
),( 00 yx
),( 1919 yx
Representation
13
C. Nikou – Image Analysis (T-14)
initial intermediate final
How should we adjust the current contour to form the new
contour at each iteration?
• Define a cost function (“energy” function) that says how
good a candidate configuration is.
• Seek next configuration that minimizes that cost function.
Fitting deformable contours
14
C. Nikou – Image Analysis (T-14)
• Representation of the contours
• Defining the energy functions
– External
– Internal
• Minimizing the energy function
• Extensions:
– Tracking
– Interactive segmentation
Aspects to consider
15
C. Nikou – Image Analysis (T-14)
The total energy (cost) of the current snake is
defined as:
externalinternaltotal EEE
A good fit between the current deformable contour
and the target shape in the image will yield a low
value for this cost function.
Internal energy: encourages prior shape preferences:
e.g., smoothness, elasticity, particular known shape.
External energy (“image” energy): encourages contour
to fit on places where image structures exist, e.g., edges.
Energy function
16
C. Nikou – Image Analysis (T-14)
• It measures how well the curve matches
the image data
• It “Attracts” the curve toward different
image features
– Edges, lines, texture gradient, etc.
External energy
17
C. Nikou – Image Analysis (T-14)
Magnitude of gradient - (Magnitude of gradient)
2 2( ) ( )x yG I G I
How do edges affect the shape
of the rubber band?
Think of external energy from
image as gravitational pull
towards areas of high contrast
External energy (cont.)
2 2( ) ( )x yG I G I
18
C. Nikou – Image Analysis (T-14)
• Gradient images and
• External energy at a point on the curve is:
• External energy for the whole curve:
),( yxGx ),( yxGy
)|)(||)(|()( 22 yxexternal GGE
21
0
2 |),(||),(| iiy
n
i
iixexternal yxGyxGE
External energy (cont.)
19
C. Nikou – Image Analysis (T-14)
• External image can instead be taken from the distance
transform of the edge image.
original - gradient
Distance transform
edges
Value at (x,y) indicates how far that position is
from the nearest edge point
Distance transform
20
C. Nikou – Image Analysis (T-14)
What are the underlying
boundaries in this fragmented
edge image?
And in this one?
Internal energy: intuition
21
C. Nikou – Image Analysis (T-14)
A priori, we want to favor smooth shapes, contours with
low curvature, contours similar to a known shape, etc.
to balance what is actually observed (i.e., in the gradient
image).
Internal energy: intuition (cont.)
22
C. Nikou – Image Analysis (T-14)
For a continuous curve, a common internal energy term
is the “bending energy”.
At some point v(s) on the curve, this is:
Tension,Elasticity
Stiffness,Curvature
sd
d
ds
dsEinternal 2
2
))((
22
Internal energy
23
C. Nikou – Image Analysis (T-14)
• For our discrete representation,
• Internal energy for the whole curve:
10),( niyx iii
i1ivds
d
11112
2
2)()( iiiiiiids
d
1
0
2
11
2
1 2n
i
iiiiiinternalE
…
Note these are derivatives relative to position---not spatial
image gradients.
Why do these reflect tension and curvature?
Internal energy
24
C. Nikou – Image Analysis (T-14)
(1,1) (1,1)
(2,2)
(3,1)(3,1)
(2,5)
2
11 2)( iiiicurvature vE 2
11
2
11 )2()2( iiiiii yyyxxx
Example (curvature)
25
C. Nikou – Image Analysis (T-14)
• Current elastic energy definition uses a discrete estimate
of the derivative:
What is the possible problem
with this definition?
2
1
1
0
2
1 )()( ii
n
i
ii yyxx
1
0
2
1
n
i
iielasticE
Penalize elasticity
26
C. Nikou – Image Analysis (T-14)
• Current elastic energy definition uses a discrete estimate
of the derivative:
1
0
2
1
n
i
iielasticE
21
0
2
1
2
1 )()(
n
i
iiii dyyxx
where d is the average distance between
pairs of points – updated at each iteration.
Instead, we may
employ:
Penalize elasticity (cont.)
27
C. Nikou – Image Analysis (T-14)
• The preferences for low-curvature, smoothness help
deal with missing data:
Illusory contours found!
Missing data
28
C. Nikou – Image Analysis (T-14)
• If the object is a smooth variation of a known shape, we can use a term that will penalize deviation from that shape:
where are the points of the known shape.
1
0
2)ˆ(n
i
iiinternalE
}ˆ{ i
Extend the internal energy by a
shape prior
29
C. Nikou – Image Analysis (T-14)
externalinternaltotal EEE
1
0
2
11
2
1 2n
i
iiiiiinternal dE
21
0
2 |),(||),(| iiy
n
i
iixexternal yxGyxGE
Total energy
30
C. Nikou – Image Analysis (T-14)
large small medium
• e.g., weight controls the penalty for internal elasticity
Fig from Y. Boykov
Total energy (cont.)
31
C. Nikou – Image Analysis (T-14)
• A simple elastic snake is defined by:
– A set of n points,
– An internal energy term (tension,
bending, plus optional shape prior)
– An external energy term (gradient-
based)
• To segment an object:
– Initialize in the vicinity of the object
– Modify the points to minimize the total
energy
Recap
32
C. Nikou – Image Analysis (T-14)
• Representation of the contours
• Defining the energy functions
– External
– Internal
• Minimizing the energy function
• Extensions:
– Tracking
– Interactive segmentation
Aspects to consider
33
C. Nikou – Image Analysis (T-14)
• Several algorithms have been proposed to
fit deformable contours.
• We’ll look at:
– Greedy search
– Dynamic programming (for 2D snakes)
– Calculus of variations
Energy minimization
34
C. Nikou – Image Analysis (T-14)
• For each point, define a search
window around it and move to where
energy function is minimal
– Typical window size, e.g., 5 x 5 pixels
• Stop when a predefined number of
points have not changed in last
iteration, or after a maximum number
of iterations
• Note:
– Convergence not guaranteed
– Need decent initialization
Energy minimization (greedy)
35
C. Nikou – Image Analysis (T-14)
1v
2v3v
4v6v
5v
• The Viterbi algorithm.
• Iterate until optimal position for each point is the center
of the box, i.e., the snake is optimal in the local search
space constrained by boxes.
[Amini, Weymouth, Jain, 1990]
Energy minimization (dynamic
programming)
36
C. Nikou – Image Analysis (T-14)
1
1
11 ),(),,(n
i
iiintotal EE
• Possible because snake energy can be rewritten as a
sum of pair-wise interaction potentials:
• Or sum of triple-interaction potentials.
1
1
111 ),,(),,(n
i
iiiintotal EE
Dynamic programming
37
C. Nikou – Image Analysis (T-14)
21
1
2
11 |),(||),(|),,,,,( iiy
n
i
iixnntotal yxGyxGyyxxE
2
1
1
1
2
1 )()( ii
n
i
ii yyxx
1
1
2
1 ||)(||),,(n
i
intotal GE
1
1
2
1 ||||n
i
ii
),(...),(),(),,( 113222111 nnnntotal vvEvvEvvEE
2
1
2
1 ||||||)(||),( iiiiii GE where
Re-writing the above with : iii yxv ,
Snake energy: pair-wise interactions
38
C. Nikou – Image Analysis (T-14)
),(...),(),( 11322211 nnntotal vvEvvEvvEE
),( 322 vvE),( 211 vvE
Main idea: determine the optimal position (state) of
predecessor, for each possible position of self.
Then backtrack from best state for last vertex.
1v 2v 3v 4v 5v
Viterbi algorithm
4 4 5( , )E v v3 3 4( , )E v v
39
C. Nikou – Image Analysis (T-14)
),( 44 nvvE),( 433 vvE
)3(3E
)(3 mE )(4 mE
)3(4E
)2(4E
)1(4E
)(mEn
)3(nE
)2(nE
)1(nE
)2(3E
)1(3E
)(2 mE
)3(2E
),(...),(),( 11322211 nnntotal vvEvvEvvEE
),( 322 vvE
)1(2E
)2(2E
),( 211 vvE
0)1(1 E
0)2(1 E
0)3(1 E
0)(1 mE
states
1
2
…
m
vert
ices
1v 2v 3v 4v nv
)( 2nmOComplexity: vs. brute force search ____?
Viterbi algorithm (cont.)
40
C. Nikou – Image Analysis (T-14)
),(...),(),( 11322211 nnn vvEvvEvvE
DP can be applied to optimize an open ended snake
For a closed snake, a “loop” is introduced into the total energy.
1n
),(),(...),(),( 111322211 vvEvvEvvEvvE nnnnn
1
n
2
1n
3 4
Work around:
1) Fix v1 and solve for rest .
2) Fix an intermediate node at
its position found in (1),
solve for rest.
Viterbi algorithm (cont.)
41
C. Nikou – Image Analysis (T-14)
Calculus of variations
• Given a 1D function ( ) :[0,1] ,u x
the basic problem is to minimize a given
energy (functional) w.r.t u(x):
1
0( ) ( , , )E u F x u u dx
with boundary conditions: (0) , (1)u a u b
2:F and given by the problem.
42
C. Nikou – Image Analysis (T-14)
Calculus of variations (cont.)
• First variation:
0 0 ( ) ( ) 0,E
E E u v E uu
:[0,1] , (0) , (1)v v a v b
v being a small increment satisfying the
boundary conditions.
1 1
0 0( ) ( ) ( , , ) ( , , )E u v E u F x u v u v dx F x u u dx
1
0[ ( , , ) ( , , )]F x u v u v F x u u dx
43
C. Nikou – Image Analysis (T-14)
Calculus of variations (cont.)
• Using Taylor series approximation:
1
00 [ ( , , ) ( , , )] 0E F x u v u v F x u u dx
( , , ) ( , , ) ...F F
F x u v u v F x u u v vu u
the energy function becomes:
1
00 0
F FE v v dx
u u
44
C. Nikou – Image Analysis (T-14)
Calculus of variations (cont.)
• Integrating the second integral by parts:
Since this holds for all v, the Euler equation is:
1 1
0 00 0
F FE v dx v dx
u u
11 1 1
0 0 00
0 0F F d F F d F
v dx v v dx v dxu u dx u u dx u
0F d F
u dx u
45
C. Nikou – Image Analysis (T-14)
Calculus of variations (cont.)
• In a similar form, for the energy:
we obtain the Euler equation:
1
0( ) ( , , , )E u F x u u u dx
2
20
F d F d F
u dx u dx u
46
C. Nikou – Image Analysis (T-14)
Calculus of variations (cont.)
• Analogous is the 2D problem:
whose Euler equation is
2 21 1
2 20 0( ) ( , , , , , )
u u u uE u F x y dxdy
x y x y
2 2
2 20
x y xx yy
F d F d F d F d F
u dx u dy u dx u dy u
47
C. Nikou – Image Analysis (T-14)
Back to the snake energy
and the Euler equation is:
221 2
2
0
( ) ( ) ( ) ( )internal image
dv d vE v E E a b cg s ds
ds ds
The image force is of the form:2
1( )
1 ( )g s
I s
2 4
2 40
d v d va b c g
ds ds
48
C. Nikou – Image Analysis (T-14)
Iterative scheme
• There is no closed form solution.
• We consider that we have an initial
solution.
• An iterative Newton-type scheme is
applied until stability is achieved:
2 4
2 4
dv d v d va b c g
dt ds ds
49
C. Nikou – Image Analysis (T-14)
Iterative scheme (cont.)
• Temporal discretization:
,
Tt t t t t tdv v v x x y y
dt
• Spatial discretization: ,
T
dv v v
ds x y
2 4
1 1 2 1 1 22 4, 2 , 4 6 4 6i i i i i i i i i i i
v v vx x x x x x x x x x
x x x
2 4
1 1 2 1 1 22 4, 2 , 4 6 4 6i i i i i i i i i i i
v v vy y y y y y y y y y
y y y
50
C. Nikou – Image Analysis (T-14)
Iterative scheme (cont.)
1
1
t t T
x
t t T
y
I A c
I A c
x x g
y y g
• Plugging the previous results into the main
equation, the problem is solved independently
in each spatial direction:
0 1 0 1 0 0 0 0,... , ,... , ( , )... , ( , )...x y
N N
T T T Tt t t t t t t t t t t t t t
x yx x y y g x y g x y
x y g g
circulant 6 2 (4 2 ) 0 0 ... 0 (4 2 )N N
A b a b a b b b a
51
C. Nikou – Image Analysis (T-14)
• Representation of the contours
• Defining the energy functions
– External
– Internal
• Minimizing the energy function
• Extensions:
– Tracking
– Interactive segmentation
Aspects to consider
52
C. Nikou – Image Analysis (T-14)
1. Use final contour/model extracted at frame t as
an initial solution for frame t+1
2. Evolve initial contour to fit exact object boundary
at frame t+1
3. Repeat, initializing with most recent frame.
Tracking Heart Ventricles (multiple frames)
Tracking by deformable contours
53
C. Nikou – Image Analysis (T-14)
Visual Dynamics Group, Dept. Engineering Science, University of Oxford.
Traffic monitoringHuman-computer interactionAnimationSurveillanceComputer assisted diagnosis in medical imaging
Applications:
Tracking by deformable contours
(cont.)
54
C. Nikou – Image Analysis (T-14)
3D active contours
55
C. Nikou – Image Analysis (T-14)
• May over-smooth the boundary
• It cannot follow topological changes of
objects
Limitations
56
C. Nikou – Image Analysis (T-14)
• External energy: snake does not really “see” object
boundaries in the image unless it gets very close to it.
image gradientsare large only directly on the boundary
I
Limitations (cont.)
• Solution: Balloon force or gradient
vector flow snake
57
C. Nikou – Image Analysis (T-14)
Gradient vector flow
• Diffusion of the edge force.
• Compute a new force field u(x,y), v(x,y)
minimizing the energy:
2 2 2 2 2 2 2 21( , ) ( ) ( ) ( ) ( )
2x y x y x y x xE u v u u v v g g u g u g dxdy
• Euler equations:
2 2 2
2 2 2
( )( ) 0
( )( ) 0
x y x
x y y
u g g u g
v g g v g
58
C. Nikou – Image Analysis (T-14)
Traditional external force field v/s GVF field
Traditional force
GVF force
(Diagrams courtesy “Snakes, shapes, gradient vector flow”, Xu, Prince)
59
C. Nikou – Image Analysis (T-14)5-59
Diffusing Image Gradients
image gradients diffused via Gradient Vector Flow (GVF)
Chenyang Xu and Jerry Prince, 98 http://iacl.ece.jhu.edu/projects/gv
f/
60
C. Nikou – Image Analysis (T-14)
More examples
61
C. Nikou – Image Analysis (T-14)
3D surfaces
62
C. Nikou – Image Analysis (T-14)
Pros:
• Useful to track and fit non-rigid shapes
• Contour remains connected
• Possible to fill in “subjective” contours
• Flexibility in how energy function is defined, weighted.
Cons:
• Must have decent initialization near true boundary, may
get stuck in local minimum
• Parameters of energy function must be set well based on
prior information
Pros and cons