spatio-temporal shape modeling and analysis · • concept: given a set of discrete shapes, ... •...
TRANSCRIPT
Spatio-Temporal Shape Modeling and Analysis
Guido Gerig
James Fishbaugh, Marcel Prastawa
Scientific Computing and Imaging Institute, University of Utah
Martin Styner, Stephen Pizer, UNC
Stanley Durrleman, Xavier Pennec, INRIA
Content
• Motivation Longitudinal Modeling
• Overview Correspondence
• Currents: Correspondence-free Modeling
• Driving Applications:
• Early Brain Development in Autism
• Huntington’s Disease (HD)
• Mandibular Growth
• Shape Analysis via Grid Deformations
• Growth Trajectories via Time Warp
WM Maturation
Longitudinal/Serial Image Data
Neo 1yr 2yr
Pediatrics: Brain Growth Aging / Neurodegeneration
Trauma: Baseline – Follow-up
Tumor Growth
• Image analysis technology for 4D data is lagging behind acquisition
• Often: individual time-point analysis, ignores causality
Jaw Growth 15-22 yrs
Spatiotemporal Modeling: Natural Task in Clinical Reasoning
Motivation:
Development, degeneration, effects of therapeutic intervention are dynamic processes.
Personalized health care: Individual trajectories compared to expected “norm”.
Clinical terminology: Atypical, Monitoring
Departure from typical development, deviation from healthy
Typical but delayed growth patterns, catch-up, atypical development
Analysis of recovery for each patient
Predict onset of clinical symptoms
Monitor efficacy of treatment
→ Focus on longitudinal design & longitudinal analysis
Shape and Registration
D’Arcy Wentworth Thompson, On Growth and Form (1917, mathematics and biology)
Why Shape?
Shape Similarity in Twins
Upper row: identical twin pairs
Lower row: non-identical twin pairs
Styner/Gerig PNAS 2005
Shape >> Volume
Example Infant Study: Cross-sectional vs. Longitudinal
A: age 2 A: age 4 B: age 2 B: age 4
Cross-sectional: Huge changes between sets of shapes
Longitudinal: Subtle changes of sets of shapes with time
Content
• Motivation Longitudinal Modeling
• Overview Correspondence
• Currents: Correspondence-free modeling
• Driving Applications: • Early Brain Development in Autism
• Huntington’s Disease (HD)
• Mandibular Growth
• Shape Analysis via Grid Deformations
• Growth Trajectories via Time Warp
Shape and Registration
Ch. G. Small, The Statistical Theory of Shape
Homology:
Corresponding
(homologous)
features in all
skull images.
Geometric Correspondence: Shapes
PDM Model
(Cootes/Taylor):
• Point to point
correspondence for
shape modeling
• PCA analysis
• Major eigenmodes
of shape variability
Correspondence through spherical
harmonic surface representations
Normalization using first order ellipsoid:
•Rotation of parameter space to align major axis
•Spatial alignment to major axes
Parameters rotated to
first order ellipsoids
Optimizing 3D Correspondences
Rhodri Davies, Carole Twining, Tim Cootes, Danny Allen, Chris Taylor: MDL Model Szekely/Brechbuehler/Styner/Gerig, SPHARM
Optimization of Correspondence in 3D and over time: Particle Systems
2 entropy terms:
• Surface entropy -> distribute the particles “evenly” across surfaces.
• Ensemble entropy -> how are particles similar across a set of surfaces.
Particle system correspondence (Cates, Whitaker et al., Utah, Styner et al., UNC): Minimization of Entropy:
Particle Correspondence Model C
onfigura
tion S
pace
S
hape S
pace
Accurate Representation (in Configuration Space)
vs.
Compact Model (in Shape Space)
Surface Entropy Ensemble Entropy
[Cates et al. IPMI 2007]
Modeling Head Shape Change
Link to Movie
Changes in head size with age
Changes in head shape with age
Datar, Cates, Fletcher, Gouttard, Gerig, Whitaker, Particle-based Shape Regression, MICCAI 2009
Medial Axis / Skeletal Representations: Intrinsic Shape Model
M-rep: Pizer et al.
(discrete)
CM-rep: Yushkevich
(continuous, parametric)
Gorczowski et al., T-PMI 2010,
Stats on deformations vs. thickness
Graph Spectra/Laplacian
Courtesy Hervé Lombaert, Jon Sporring, Kaleem Siddiqi, Mc Gill, IPMI 2013
Graph Spectra/Laplacian
Courtesy Hervé Lombaert, Jon Sporring, Kaleem Siddiqi, Mc Gill, IPMI 2013
Discussion Correspondence
• Automatic optimization of correspondences efficient and robust for simple 3D shapes.
• PDM and Entropy Methods require fixed topology: Time-consuming pre-processing.
• Challenges: – thin, spiky objects
– gaps, missing parts
Content
• Motivation Longitudinal Modeling
• Overview Correspondence
• Currents: Correspondence-free Modeling
• Driving Applications: • Early Brain Development in Autism
• Huntington’s Disease (HD)
• Mandibular Growth
• Shape Analysis via Grid Deformations
• Growth Trajectories via Time Warp
Problem: Correspondences between Shapes with Variable Topology
Brain ventricles for infants 6mo to 2yrs DTI Fiber Tracts from two subjects
movie
Durrleman, Pennec, Ayache et al.
“Correspondence-free” Registration: Currents
Topology and shape differences and noise can make point-to-point correspondence hard:
Currents: Objects that integrate vector fields
Shape: Oriented points = Set of normals (tangents)
Distance between curves:
[Glaunes2004] Glaunes, J., Trouve, A., Younes, L. Diffeomorphic matching of distributions: a new approach, … CVPR 2004.
[Durrleman2008] S. Durrleman, X. Pennec, A. Trouvé, P. Thompson, N. Ayache, Inferring Brain Variability from Diffeomorphic
Deformations of Currents: an integrative approach, Medical Image Analysis 2008
Currents integrate vector field:
•W: test space of vector fields (Hilbert space)
•W*: the space of continuous maps W->R
•W* includes smooth curves, polygonal lines, surfaces,
meshes.
[Vaillant and Glaunès IPMI’05, Glaunès PhD’06]
Durrleman, PhD thesis, 2010
Durrleman, PhD thesis, 2010
Durrleman, PhD thesis, 2010
Durrleman, PhD thesis, 2010
Distance between shapes:
• No point correspondence
• No individual line correspondence
• Robust to line interruption
• Need consistent orientation of lines/surfaces
• Is a norm
The space of currents: a vector space:
• Addition = union
• Scaling = weighting different structures
• Sign = orientation
Durrleman, PhD thesis, 2010
Shape Regression w/o Correspondence
• Regression model:
• Variational problem, minimize:
d: modeling shapes as “currents” → distance metric
• → From sampled to continuous shape model:
shape baseline :
ndeformatio varyingtime:
)()(
0
0
M
MtSS
t
tii
Vaillant, M., Glaunes, J.: Surface matching via currents (IPMI 2005). Springer LNCS Vol. 3565) 381–392 Durrleman, S., Pennec, X., Trouve, A., Thompson, P., Ayache, N.: Inferring brain variability from diffeomorphic deformations of currents: an integrative approach. Medical Image Analysis 12/5(12) (2008) 626–637
ii MtS )(
• Concept: Given a set of discrete shapes, interpolate a continuous 4D
growth model via shape regression.
• Assumption: Growth/degeneration of biological tissue is inherently
smooth in space and time & nonlinear, locally varying process.
• Method: Continuous flow of diffeomorphisms via correspondence-free
“currents”. Cost function = Data Matching + Regularity.
Durrleman, Pennec, Ayache, Trouve, Gerig, MICCAI ‘09
Fishbaugh, Durrleman, Gerig, MICCAI ’11, SPIE’12, MICCAI’12, IPMI’13
4D Shape Modeling from Time-Discrete Data
Acceleration Controlled Shape Regression
We define the acceleration field a(x(t)) as a vector field of the form
the shape points carrying a point force vector αi
a Gaussian
kernel with standard deviation λV
Time varying deformation given by:
initial position
initial velocity
Regression Criterion
Acceleration Controlled Shape Regression
Point forces α Acceleration Velocity
Evolution of cerebellum from 6 to 24 months
Piecewise Geodesic vs Acceleration Controlled
Synthetic experiment comparing piecewise geodesic and acceleration controlled shape regression
Piecewise geodesic Acceleration controlled
Interpolation Properties
Summary
Benefits:
● More biologically realistic trajectories ● Nice interpolation properties
Drawbacks:
● Not compact or generative
Content
• Motivation Longitudinal Modeling
• Overview Correspondence
• Currents: Correspondence-free modeling
• Driving Applications:
• Early Brain Development in Autism
• Huntington’s Disease (HD)
• Mandibular Growth
• Shape Analysis via Grid Deformations
• Growth Trajectories via Time Warp
Autism: Longitudinal Infant Neuroimaging Study
ACE-IBIS NIH Study: UNC (PI), McGill, Seattle, WU, CHOP, Utah (Image Analysis)
Brain enlargement in autism starts at year 1.
Why? What? Effect?: Longitudinal MRI/DTI study w. >1250 MRI/DTI
Better understanding → Early intervention to improve outcome
30
35
40
45
50
55
0 3 6 9 12 15 18 21 24 27 30 33 36
Age (months)
Hea
d C
irc
um
fere
nc
e
Combined Controls
Autism
N= 113, ADI, ADOS N= 189 community controls
Brain Enlargement on MRI
Longitudinal Head Circumference (Cody
Hazlett et al., ArchGen Psyc Dec. 2005)
Longitudinal Shape Regression
Durrleman, Fishbaugh, Gerig, MICCAI 2011, MICCAI 2012
movie
4D Shape Regression
From discrete 3D shapes (6m,12m,24m) to continuous 4D shape model
Fishbaugh, Durrleman, Gerig, MICCAI 2011, 2012
6 12 24 Time (months)
Individual 4D Growth Profiles
HR+
HR-
LR-
Longitudinal Shape Modeling
Individual 4D model
Work in progress: Stats of 4D growth profiles
Autism Research Collaboration UNC (Piven, Hazlett)
HR+: High risk infant ADOS pos.
HR-: High risk infants ADOS neg.
LR-: Low risk healthy infants
Autism Research Collaboration ACE-IBIS (PI J. Piven, UNC)
Motivation: Longitudinal Imaging in HD Searching for noninvasive biomarker with imaging…
• Symptomatic HD imaging findings
– Atrophied caudate and putamen
– Disproportionate loss of white matter
• Prodromal HD imaging findings
– Striatal atrophy correlates with:
• Neurological impairment
• Poorer performance on cognitive assessments
• Years to motor symptom onset
– Decreased white matter volume
• Seen >15 years before symptom onset
• Morphological study: cognitive deficits had stronger correlation with cerebral white matter atrophy than striatal atrophy Courtesy Jane Paulsen, Hans Johnson, U-Iowa
Motivation: Huntington’s Disease
Courtesy Prof. Hans Johnsen, IOWA
NIH PREDICT Study: Longitudinal imaging study (3-5 scans over 2yr intervals).
Relationship between estimated years to diagnosis of Huntington's disease and
motor exam score and striatal volumes. Distribution of age of onset for individuals
with 36-56 CAG repeats based on the parametric model. Red indicates most likely
time of diagnosis. Blue line is proposed time period when interventional therapies
would have greatest impact.
PREDICT-HD: Longitudinal Imaging Study
58yrs 59yrs 60yrs
Huntington’s Disease: • Neurodegenerative, progressive disease Visual assessment: Subtle changes over time: • Requires well-calibrated image data • Precise, robust segmentations
Huntington’s Disease: Joint analysis of sets of anatomical structures
• Data: Iowa Huntington Disease (HD) study (NAMIC)
• Goal: Prediction of onset of HD from longitudinal preclinical imaging
Clinical Application: Neurodegeneration
in Huntington’s Disease
Continuous individual
subjects’ growth models
Colllaboration Hans Johnson, U-Iowa
Quantitative information
derived from 4D shapes
Muralidharan, Fletcher, Gerig, 2013
Problem for Personalized Profiles: Variability in 3D Segmentation
Volumes from one subject
Volumes from multiple subjects
with varying disease burden
HD: Joint 4D Modeling of subcortical structures
Subject-Specific Shape Modeling
Caudate volume for 32 subjects (3 time pts) extracted after shape regression.
Observed volumes are shown as circles, which highlight the noise in segmentation.
We estimate consistent shape trajectories by considering all shapes simultaneously
which respects the interplay between shape boundaries and locations.
Muralidharan et al, MICCAI 2014
National Alliance for Medical Image Computing
http://na-mic.org
Longitudinal Shape Modeling
Example mandibular surgery case (L. Cevidanes, NA-MIC ancillary
grant).
• V2- 16 years of age before jaw surgery
in the upper jaw maxilla
• V6- 16 years of age 2 months post
surgery
• V10- 18 years of age, 2 years post-
surgery
• V12- 22 years of age, 6 years post-
surgery
National Alliance for Medical Image Computing
http://na-mic.org
Longitudinal Shape Modeling
Example mandibular surgery case (L. Cevidanes, NA-MIC ancillary
grant).
EXOSHAPE ACCEL Tool: Correspondence-free, controlled
acceleration, no tuning, 20’ computing time
Color: speed:
blue = slow and red = fast
Application: Craniosynostosis
4D atlas from 350 full brain shapes (6 – 825 days)
Paniagua, B. et al. 3D of brain shape and volume after cranial vault remodeling surgery for Craniosynostosis correction in infants. SPIE Medical Imaging 2013
velocity
Content
• Motivation Longitudinal Modeling
• Overview Correspondence
• Currents: Correspondence-free modeling
• Driving Applications:
• Early Brain Development in Autism
• Huntington’s Disease (HD)
• Mandibular Growth
• Shape Analysis via Grid Deformations
• Growth Trajectories via Time Warp
Shape Analysis via Transformations
D’Arcy Wentworth Thompson, On Growth and Form (1917, mathematics and biology)
GEODESIC REGRESSION OF IMAGE AND SHAPE DATA
Fishbaugh et al, ISBI 2014
GEODESIC REGRESSION OF IMAGE AND SHAPE DATA
Fishbaugh et al, ISBI 2014
Pediatric Brain Development
1) Estimation with images only
2) Estimation jointly but only showing image
3) Estimation jointly and showing both image and white matter
4) Estimation with white matter surfaces only
Pediatric Brain Development
Pediatric Brain Development
Fishbaugh et al, ISBI 2014
Huntington’s Disease: Joint 4D Modeling of Shapes and Images
Subject-specific 4D shape & image regression
Control 2yrs Interval Huntington’s D. 2yrs Interval
Fishbaugh et al., IPMI 2013
Work in Progress: Patient-specific 4D
shape & image regression Control Extrapolated HD Extrapolated
interpolation extrapolation time
Fishbaugh et al., ISBI ‘13, IPMI ‘13
Deformetrics with Sparsity: Tackling fundamental problem of high-dim
features & low-dim sample size (HDLSS)
Image evolution described by considerably fewer parameters, Concentrated in areas undergoing most dynamic changes
Durrleman, 2013 / Fishbaugh IPMI 2013, GSI 2013
Initialization Atlas
Statistics
Common template for
8 Down’s syndrome patients + 8 Ctrls
Most discriminative axis
Classification (leave-2-out) with 105 control points:
specificity sensitivity
Max Likelihood 100% (64/64) 100% (64/64)
LDA 98% (63/64) 100% (64/64)
Importance of optimization in control points positions!
Classification (leave-2-out) with 8 control points:
specificity sensitivity
Max Likelihood 97% (62/64) 100% (64/64)
LDA 94% (60/64) 89% (57/64)
Content
• Motivation Longitudinal Modeling
• Overview Correspondence
• Currents: Correspondence-free modeling
• Driving Applications: • Early Brain Development in Autism
• Huntington’s Disease (HD)
• Mandibular Growth
• Shape Analysis via Grid Deformations
• Growth Trajectories via Time Warp
4D Shape Analysis: Spatiotemporal Registration
Comparing two evolutions: Align shape trajectories:
Spatiotemporal variability analysis of longitudinal shape data, S. Durrleman, X. Pennec, A. Trouvé, G. Gerig, N. Ayache, MICCAI’10, IJCV 2012
ii MtS )(
ii NtT )(
Formalism captures: • continuous regression: ϕ(t) • morphological change: χ • change of growth speed: ѱ(t)
)()( 0MtS t
function change time:
ndeformatio lgeometrica :
where))),((()(
tStT
4D Shape Analysis: Spatiotemporal Registration
Spatiotemporal variability analysis of longitudinal shape data, S. Durrleman, X. Pennec, A. Trouvé, G. Gerig, N. Ayache
4D Shape Analysis: Spatiotemporal Registration
Spatiotemporal Variability: • morphological changes • change of growth speed
Spatiotemporal variability analysis of longitudinal shape data, S. Durrleman, X. Pennec, A. Trouvé, G. Gerig, N. Ayache
(Homo habilis to sapiens vs. homo erectus)
Preliminary Results: Amygdala Growth in Autism
Results: Amygdala Growth in Autism
Conclusions
• Spatio-temporal Image & Shape Analysis: Emerging field with
new fundamental problems (math, stats, imaging, modeling):
– Multidisciplinary by definition.
– Actively developing field driven by new imaging technologies and
novel biomedical driving problems.
– Challenging fundamental, algorithmic and statistical problems.
– Research progress enables new scientific discoveries.
• Main driving motivation: Trajectory of change vs. Cross-
sectional comparisons.
• Clinically highly relevant for quantitative analysis of
longitudinal image data.
Acknowledgements • NIH-NINDS: 1 U01 NS082086-01: 4D Shape Analysis
• NIH-NIBIB: 2U54EB005149-06 , NA-MIC: National Alliance for MIC
• NIH (NICHD) 2 R01 HD055741-06: ACE-IBIS (Autism Center)
• NIH NIBIB 1R01EB014346-01: ITK-SNAP
• NIH NINDS R01 HD067731-01A1: Down’s Syndrome
• NIH P01 DA022446-011: Neurobiological Consequences of Cocaine Use
• USTAR: The Utah Science Technology and Research initiative at the Univ. of Utah
• UofU SCI Institute: Imaging Research Team
• Insight Toolkit ITK
Acknowledgements Methodology Development:
• James Fishbaugh, Utah
• Stanley Durrleman, INRIA Paris
• Xavier Pennec, INRIA Sophia Antipolis
• Ross Whitaker, Utah
• Martin Styner, UNC
Clinical Longitudinal Imaging:
• Joseph Piven, UNC Psychiatry
• Jane Paulsen and Hans Johnson, U of Iowa
• Lucia Cevidanes, UMICH
National Alliance for Medical Image Computing
http://na-mic.org
Freely available Software
ExoshapeAccel: C/C++ NAMIC toolkit SW
for estimating continuous evolution from a
discrete collection of shapes,
James Fishbaugh Public download
Stanley Durrleman
http://www.deformetrica.org/