STATISTICAL ANALYSIS OF SHAPES OF 2D CURVES, 3DCURVES & FACIAL SURFACES
Anuj SrivastavaDepartment of Statistics
Florida State University
This research supported in part by ARO and NSF.
Presented at IMA Workshop on Shape Spaces, April 4th, 2006.
COLLABORATORS
• Mathematics: Eric Klassen, Washington Mio, Dave Wilson (Univ of
Florida)
• Computer Science: Mohamed Daoudi (ENIC, France), Chafik Samir
(grad, ENIC, France), Aastha Jain (undergrad, IIT Delhi)
• Statistics: Dave Kaziska (AFIT), Sanjay Saini (grad)
• Electrical Engineering: Shantanu Joshi (grad)
• Neuroscience: Zhaohua Ding and Chunming Li (Vanderbilt Inst. of
Imag. Sc.)
TENETS OF GRENANDER’S PATTERN THEORY
General Pattern Theory, Oxford University Press, 1993, In Introduction:
1. Create representations in terms of algebraic systems with probabilistic
superstructures.
What probabilistic superstructures should we have on our shape spaces?
2. Analyze structures from the perspective of ....statistical inference.
How can we generate inferences on such nonlinear, high-dimensional
space?
3. Apply ... to particular applications.
Are we there yet?
TOPICS COVERED
• Closed Curves in R2
– Geodesics Using Shooting Method
– Statistics, Bayesian Analysis
– Stochastic Processes on Shape Spaces
• Closed Curves in R3
– Geodesics Using Path-Straightening Method
• Shapes of Facial Surfaces
– Representations Using Facial Curves
– Face Recognition
• Summary
PAST WORK: PLANAR CURVES
1. Representations of curves: Assuming arc-length parametrization.
0 1 2 3 4 5 6 7−2
−1
0
1
2
3
4
5
6
7
Original Shape Angle Function θ
0 1 2 3 4 5 6 7−30
−20
−10
0
10
20
30
0 1 2 3 4 5 6 7−1.5
−1
−0.5
0
0.5
1
1.5
2
Curvature Function Coordinate Functions
C = {θ| 1
2π
Z 2π
0
θ(s)ds = π,
Z 2π
0
cos(θ(s))ds = 0,
Z 2π
0
sin(θ(s))ds = 0}
2. Shape-preserving transformations:
Automatically: Rotation, translation, and scaling.
Manually: Placement of origin, i.e. registration group is S1.
0 10 20 30 40 50 60 70 80 90 100−2
−1
0
1
2
3
4
5
6
7
8
0 10 20 30 40 50 60 70 80 90 100−1
0
1
2
3
4
5
6
7
Shape space S = C/S1.
3. Metric: L2 metric on angle functions
For g, h ∈ Tθ(S), we define:
〈g, h〉 =∫ 2π
0
g(s)h(s)ds .
4. Geodesics: Constructed numerically using shooting method. Paths
correspond to minimum bending energy between shapes.
0 2 4 6 8 10 12
0
1
2
0 2 4 6 8 10 12
0
1
2
5. References : Klassen et al., (Asilomar Conference (2002), EMMCVPR
(2003), and IEEE PAMI (March 2004)).
PAST WORK: ELASTIC PLANAR CURVES
1. Representations: Shapes are now represented by a pair of functions:
log-speed (φ) and angle (θ). eφ(s) provides speed of traversal along the
curve.
∫exp(φ)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
C = {(φ, θ)| 1
2π
Z 2π
0
θ(s)eφ(s)ds = π,
Z 2π
0
cos(θ(s))eφ(s)ds = 0,
Z 2π
0
sin(θ(s))eφ(s)ds = 0}
2. Shape-preserving transformations:
Automatic: Rotation, translation, and scaling.
Manually: (i) Placement of origin. (ii) Variable-speed parametrization, i.e.
re-parametrization group D: for γ ∈ D,
(φ, θ) ◦ γ = (φ ◦ γ + log(γ′), θ ◦ γ) .Shape space S = C/(S1 ×D).
Dynamic programming is used to match shapes.
3. Metric: L2 metric on angle and log-speed functions
For (g1, h1), (g2, h2) ∈ T(φ,θ)(S), we define:
〈(g1, h1, (g2, h2)〉 = a
∫ 2π
0
g1(s)g2(s)eφ(s)ds+b∫ 2π
0
h1(s)h2(s)eφ(s)ds .
4. Geodesics: Constructed numerically using shooting method. Paths
correspond to optimal bending and stretching/compression between
shapes.
Top row: non-elastic representation. Bottom row: elastic representation.
5. References : Mio et al., (CVPR 2004, IJCV in review).
SHAPE CLUSTERING
613 21
2344
7 14 24 25 36 39
918
2242
10 16 19 30 3345 47 48
1117
28 31
34 37 41 49
8 1526
3240
434612 20 27 29 35 38
50
1 2 3 4 5
Minimization of cumulative dispersion within classes using Metropolis-based
simulated annealing
References : Joshi et al., (SSP 2003, ECCV 2004, IEEE PAMI April 2005).
STATISTICS OF ELASTIC SHAPES
Examples of observed shapes
Intrinsic definition of mean shape: d(·, ·) denotes geodesic lengths.
µ = argminθ∈S
∫Sd(θ, φ)2f(φ)dφ .
TANGENT PRINCIPAL COMPONENTS (TPCA)
• Let Tµ(S) be the space of vectors tangent to S at µ ∈ S . Geodesics
are used to map shapes from S to Tµ(S).
• S is a nonlinear manifold but Tµ(S) is a vector space. Derive probability
models in this space. For example, use PCA to reduce dimension and
impose models on coefficients.
PROBABILITY MODELS ON TPCs
1. Non-parametric models:
• Let f(1)j , j = 1, . . . , d be the density estimate of aj , the jth TPC of
the shape α.
• Assuming independence of TPCs, we obtain:
f (1)(α) =∏dj=1 fj(aj)
2. Gaussian Models:
• Let Σ ∈ Rd×d be the diagonal matrix in SVD of K .
• The Gaussian model is given as, f (2)(α) =∏dj=1 h(aj ; 0,Σjj),
where h(y; z, σ2) ≡ 1√2πσ2 exp(−(y − z)2/(2σ2))
3. Gaussian Mixture models:
Define the mixture density
f (3)j(α) =
∏dj=1
(∑Kk=1 pkh(aj ; zk, σ
2k)
), and
∑k pk = 1.
Estimate f (3) using EM algorithm
• Density Estimates:
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.40
0.5
1
1.5
2
2.5
−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.80
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.40
0.5
1
1.5
2
2.5
−0.6 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.30
1
2
3
4
5
6
−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.20
1
2
3
4
5
6
−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.50
0.5
1
1.5
2
2.5
3
3.5
4
4.5
−0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.40
0.5
1
1.5
2
2.5
3
3.5
4
4.5
−0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.50
0.5
1
1.5
2
2.5
3
3.5
4
−0.2 −0.1 0 0.1 0.2 0.3 0.4 0.50
1
2
3
4
5
6
−0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.40
0.5
1
1.5
2
2.5
3
3.5
4
−0.2 −0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.2 0.250
1
2
3
4
5
6
Dogs Pears Mugs
• Likelihood Ratio Tests: Gaussian mixture outperforms others.
f(1)(α)f(3)(α)
5 10 15 20 25 30 35 400
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
5 10 15 20 25 30 35 400
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
5 10 15 20 25 30 35 400
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
f(3)(α)f(2)(α)
5 10 15 20 25 30 35 400
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
5 10 15 20 25 30 35 400
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
5 10 15 20 25 30 35 400
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
• References : Srivastava et al. (PAMI, April 2005, ACCV 2006).
BAYESIAN SHAPE ESTIMATION
MAP estimate of a closed curve, under a given prior model, by minimizing:
Etotal(α, I) = Eimage(α, I) + Esmooth(α) + Eprior(θ) .
Image I GVF of I Prior Mean
Evolution of curve under a gradient flow.
Reference: Joshi et al. (ECCV Workshop, 2006)
BAYESIAN SHAPE COMPLETION
Top row: Given images; Middle Row: Estimated shape using our approach;
Bottom row: Human drawing
EXTRACTION OF CARDIAC CURVES IN ULTRASOUND
(With Dave Wilson, UF)
ECG images of heart with expert generated contours for cardiac layers
overlaid.
Simply compute a geodesic path between the given shapes and overlay
them on the intermediate images.
References : Joshi et al. (IPMI, 2005).
Use the Bayesian approach to improve upon the estimation.
References : Joshi et al. (ECCV Workshop, 2006).
BAYESIAN SHAPE MODEL SELECTION
Approach: Shape is considered as nuisance variable in recognition
(Grenander et al. IEEE IT 2001):
....
...
0
5
10
15
20
25
30
35
40
models
E total
q1 q
3 q
5 q
7 q
9 q
4r q
6r q
8r q
2r
CYCLOSTATIONARY PROCESSES ON SHAPE SPACES
• Representation of human gait as a stochastic process X(t) on C.
• Cyclo-stationary process: statistics are periodic
• Metric for comparing gaits:
dp(X,Y ) = argminκ∈[0,τy ],ψ
(∫ τx
0
d(E[X(t)], E[Y (κ+ ψ(t))])2dt),
– E[·] is the expectation operator.
– κ, ψ are used to register the two gait cycles. ψ is solved using dynamic
programming.
– d(·, ·)2 is the elastic geodesic distance.
– Detection of gait cycles is automatic.
• Experimental Results: 26 subjects, different test and training data
i nearest neighbor 1 2 3 4 5 6 7 8 9 10
Elastic Shape Model 17 20 21 21 21 22 22 23 24 24
Mean-Shape Approach 13 14 16 19 19 20 20 20 20 22
Landmark-Based Approach 10 11 14 17 19 19 20 20 21 22
• References: Kaziska et al (PhD Thesis 2005, ECCV 2006).
3D FACE RECOGNITION
• We want to statistically analyze shapes of facial surfaces, i.e. define
geodesics, means, covariances, etc.
• Not as simple as curves, as there is no natural ordering of points.
• We chose a specific parametrization of facial surfaces.
3D FACE RECOGNITION USING FACIAL CURVES
• Assume that gaze direction is aligned with the z-direction. Let F be the
depth function on a facial surface.
• Let cλ be the level-curve of F for value λ. It has a shape component
and other variables (scale, rotation, etc).
• A surface S is represented by a path, parameterized by λ, on C(ignoring level curves that are not simple, closed curves).
• Examples:
• Metric: d(S1, S2) =∫λd(c1λ, c
2λ)dλ.
• Recognition rates: ∼ 60 people, 360 scans
1 1.5 2 2.5 3 3.5 4 4.5 530
40
50
60
70
80
90
Facial expressions per class used in training data sets
Rec
ogni
tion
Rat
e %
C5EucC4EucC53ucC2EucC1Euc
1 1.5 2 2.5 3 3.5 4 4.5 530
40
50
60
70
80
90
100
Facial expressions per class used in training data sets
Rec
ogni
tion
Rat
e %
C5GeoC4GeoC3GeoC2GeoC1Geo
1 1.5 2 2.5 3 3.5 4 4.5 560
65
70
75
80
85
90
Number of facial curves
Rec
ogni
tion
Rat
e %
C42Geo
C42Euc
• References: Samir et al (ICASSP 2006, PAMI in review).
INVARIANT FACIAL CURVES
• Define a function F (x) = d(x, p) where p is the tip of nose, and d(·, ·)is the geodesic distance on a facial surface. (Bronstein et al.IJCV 2005).
• Define facial curves to be the level curves of F .
• Compare surfaces by comparing their corresponding facial curves.
Facial curves are now closed curves in R3.
REPRESENTATION OF CLOSED CURVES in R3
• Let p : [0, 2π) �→ R3 be a C1-curve of length 2π,
parameterized by the arc length (can be relaxed as earlier).
• For v(s) ≡ p(s) ∈ R3, we have ‖v(s)‖ = 1 for all s ∈ [0, 2π).
• The function v is called the direction function of p and itself can be
viewed as a curve on the unit sphere S2, i.e. v : [0, 2π) �→ S
2.
REPRESENTATION OF CLOSED CURVES
• P be the set of all continuous maps from [0, 2π) → S2.
• Since we are interested in the shapes of closed curves, and we establish
that subset as follows. Define a map φ : P �→ R3 by
φ(v) =∫ 2π
0v(s)ds, and define
C = φ−1(0) ≡ {v ∈ P|φ(v) = 0} ⊂ P .
• Elements of C denote closed curves in R3.
• Inner product on Tv(C): for f, g ∈ Tv(C),
〈f, g〉 =∫ 2π
0
(f(s) · g(s))ds .
GEODESICS: A PATH STRAIGHTENING APPROACH
• Define Space: Let C be the space of all closed curves in R3. Let H be
the set of all paths in C, parameterized by t ∈ [0, 1].
• Initialize Path: For any two curves, denoted by v0 and v1, let α be a path
connecting them in C. That is, α : [0, 1] �→ C, such that α(0) = v0
and α(1) = v1.
• Path Energy: Let H0 be all such paths between v0 and v1. Define an
energy E on H0 whose critical point (gradient is zero) is a geodesic
path.
• Path-Straightening: Update α according to (negative) gradient of E until
convergence.
• Shape Space: Define a shape space S = C/T , and construct
geodesics in S .
PATH ENERGY AND ITS GRADIENT
• For a path α on H, it path energy is given by:
E(α) ≡ 12
∫ 1
0
⟨dα
dt,dα
dt
⟩dt .
(A critical point of E is a geodesic on C.)
• Goal: Our goal is to find the minimizer:
α = argminα∈H0
E(α) .
We will use a gradient approach to find this minimizer.
• To make H a Riemannian manifold, we use the Palais metric: for w1,
w2 ∈ Tα(H),
〈〈w1, w2〉〉 = 〈w1(0), w2(0)〉 +∫ 1
0
⟨Dw1
dt(t),
Dw2
dt(t)
⟩dt ,
Theorem 1 Let α : [0, 1] �→ C be a path such that α ∈ H0. Then, with
respect to the Palais metric:
1. The gradient of the energy function E on H is the vector field q along α
satisfying q(0) = 0 and Dqdt = dα
dt . That is, q is a covariant integral ofdαdt .
2. The gradient of the energy function E restricted to H0 is
w(t) = q(t) − tq(t), where q is the vector field defined in the previous
item, and q is the vector field obtained by parallel translating q(1)backwards along α.
At the critical point, we have q(t) = tq(t). Then, q(t) is covariantly
constant, q(t) = tq(t) is covariantly linear, and hence, dαdt is covariantly
constant.
References: Klassen et al (ECCV 2006, SIAM J. Computation, in review).
EXPERIMENTAL RESULTS
Experiment 1: Let the two curves of interest be:
p1(t) = (a cos(t), b sin(t), c√b2 − a2 sin2(t))
p2(t) = (a(1 + cos(t)), sin(t), 2 sin(t/2))
−1
−0.5
0
−1
−0.5
0
0.5
1
−0.15−0.1
−0.050
−1.5
−1
−0.5
0
−0.5
0
0.5
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1 1.5 2 2.5 3 3.5 4 4.5 5
3.45
3.5
3.55
3.6
Geodesic Path:
0
2
4
6
8
10
12
14
16
−2
0
2
−1
0
1
02
46
810
1214
16
−2
−1
0
1
2
−101
Experiment 2
0
0.2
0.4
0.6
0.8
1
1.2−0.2
0
0.2
0.4
0.6
0.8
1
−1
−0.5
0
−0.2
0
0.2
0.4
0.6
0.8
1
1.2 0
0.2
0.4
0.6
0.8
1
1.2
−0.8−0.6−0.4−0.2
00.2
0 2 4 6 8 10 12 1411
12
13
14
15
16
17
18
19
20
21
0 2 4 6 8 10 12 14 16
−2
−1
0
0 2 4 6 8 10 12 14 16
−1
0
1
2
FACIAL CURVES
Very preliminary:
−0.3
−0.2
−0.1
0
0.1
0.2
0.3−0.4
−0.3
−0.2
−0.1
0
0.1
0.2
0.3
0.40.05
0.1
0.15
0.2
0.25
0.3
0.35
−0.4−0.2
00.2
0.4 −0.4
−0.2
0
0.2
0.4
0.1
0.15
0.2
0.25
0.3
0.35
0
2
4
6
8
10
12
−3
−2
−1
0
1
−1
0
1
SUMMARY
• Motivated of analysis of shapes of curves.
• Presented a representation of planar shapes that is invariant to rotation,
translation, and scaling. Showed examples of computing geodesic paths
in shape spaces.
• Presented examples of statistical analysis of shapes in R2, and imposed
probability models on shapes using TPCA.
• Demonstrated an application of probability models in Bayesian shape
extraction.
• Shapes of curves in R3: mentioned a path-straightening mechanism for
constructing geodesics.
• Used analysis of 3D curves to propose a method for studying shapes of
facial surfaces.