knowledge driven segmentation introduction of cardiovascular...

Knowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006

Knowledge driven segmentationKnowledge driven segmentationof cardiovascular imagesof cardiovascular images

Boudewijn Lelieveldt, PhDDivision of Image Processing,

dept of Radiology, Leiden University Medical Center


IntroductionIntroduction

• Introduction: why prior knowledge?

• Knowledge guided image processing • Model driven• Data driven• High level reasoning

• Future directions

•• Introduction: why prior knowledge?Introduction: why prior knowledge?

•• Knowledge guided image processing Knowledge guided image processing •• Model drivenModel driven•• Data drivenData driven•• High level reasoningHigh level reasoning

•• Future directionsFuture directions

Knowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006 Knowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006

Flow

Problem: amount of dataProblem: amount of data

Function

PerfusionInfarct

Imaging

Rest Stress

Anatomy


Goal: automated segmentationGoal: automated segmentation

• Less work!!• More objective

• Less difference between observers• Less difference in repeated measurements

• More reproducible• Easier to compare with others• Easier to perform follow ups• Less patients in trial

•• Less work!!Less work!!•• More objectiveMore objective

•• Less difference between observersLess difference between observers•• Less difference in repeated measurementsLess difference in repeated measurements

•• More reproducibleMore reproducible•• Easier to compare with othersEasier to compare with others•• Easier to perform follow upsEasier to perform follow ups•• Less patients in trialLess patients in trial


Introduction: why knowledge?Introduction: why knowledge?


Why knowledge?Why knowledge?• Robust segmentation only possible using

knowledge about:• Anatomical shape + shape variation +

motion• Spatial context of organs• Intensity and data characteristics• Behavior of your algorithms

• Modality independent:• Current applications:

• Cardiac MR• Cardiac CT• LV Angiography• Echocardiography

•• Robust segmentation only possible using Robust segmentation only possible using knowledge about:knowledge about:•• Anatomical shape + shape variation + Anatomical shape + shape variation +

motionmotion•• Spatial context of organsSpatial context of organs•• Intensity and data characteristicsIntensity and data characteristics•• Behavior of your algorithmsBehavior of your algorithms

•• Modality independent:Modality independent:•• Current applications:Current applications:

•• Cardiac MRCardiac MR•• Cardiac CTCardiac CT•• LV LV AngiographyAngiography•• EchocardiographyEchocardiography


General approachGeneral approach

• Tackle KGIP from three angles:• Model driven

• Statistical shape models– Segmentation– Diagnosis

• 3D thorax template• Data driven

• Autonomous vehicle• High level reasoning

• Multi-agent systems• Data fusion

•• Tackle KGIP from three angles:Tackle KGIP from three angles:•• Model drivenModel driven

•• Statistical shape modelsStatistical shape models– Segmentation– Diagnosis

•• 3D thorax template3D thorax template•• Data drivenData driven

•• Autonomous vehicleAutonomous vehicle•• High level reasoning High level reasoning

•• MultiMulti--agent systemsagent systems•• Data fusionData fusion














Active Shape ModelsActive Shape Models

• Describe the shape of an organ in a population as• an average shape• a small number of characteristic shape

variations

• DEMO

•• Describe the shape of an organ in a Describe the shape of an organ in a population aspopulation as•• an average shapean average shape•• a small number of characteristic shape a small number of characteristic shape

variationsvariations

•• DEMODEMO


AAM Model GenerationAAM Model Generation

Set of examples (N=20)

Appearance eigenvariations

Average Shape Average patch+ eigenvariations


Subsets: Male vs. femaleSubsets: Male vs. female

Male (N=14) Female (N=6)


Subsets: Ass. professors vs. othersSubsets: Ass. professors vs. others

Ass. Professor (N=7) Non – ass. professor (N=14)


Active Appearance ModelsActive Appearance Models

• AAM’s can be used for contour tracing by:• initially positioning the model• fitting the model to the underlying image

along ‘statistically plausible’ deformations

•• AAM’sAAM’s can be used for contour tracing can be used for contour tracing by:by:•• initially positioning the modelinitially positioning the model•• fitting the model to the underlying image fitting the model to the underlying image

along ‘statistically plausible’ deformationsalong ‘statistically plausible’ deformations


AAM MatchingAAM Matching


Matching AAM’sMatching AAM’s

• To match an AAM to an image requires:• a criterion function e:

• the RMS error of the ‘difference image’ between the model and the underlying image patch

• a minimization procedure (Levenberg Marquardt / simplex)• derivatives of the criterion function with respect to

all ‘optimizable’ parameters • can be estimated using multiple linear regression• examples of derivative images

•• To match an AAM to an image requires:To match an AAM to an image requires:•• a criterion function a criterion function ee::

•• the RMS error of the ‘difference image’ between the the RMS error of the ‘difference image’ between the model and the underlying image patchmodel and the underlying image patch

•• a minimization procedure a minimization procedure (Levenberg Marquardt / simplex)(Levenberg Marquardt / simplex)•• derivatives of the criterion function with respect to derivatives of the criterion function with respect to

all ‘optimizable’ parameters all ‘optimizable’ parameters •• can be estimated using multiple linear regressioncan be estimated using multiple linear regression•• examples of derivative imagesexamples of derivative images


Variation 1 Variation 3Variation 2

Average


2D2D


2D + time modeling2D + time modeling

• Modeling a heartbeat• Define point correspondence in 2D• Define “time-correspondence”

• Divide interval between ED-ES-ED in fixed # steps• Interpolate in time (nearest neighbor)

•• Modeling a heartbeatModeling a heartbeat•• Define point correspondence in 2DDefine point correspondence in 2D•• Define “timeDefine “time--correspondence”correspondence”

•• Divide interval between EDDivide interval between ED--ESES--ED in fixed # stepsED in fixed # steps•• Interpolate in time (nearest neighbor)Interpolate in time (nearest neighbor)


Movie: Bosch e.a.Movie: Bosch e.a.


Echocardiography

• Model initialization on average pose of training set

• Fully automated match for total sequence at once

Hans Bosch (TMI 2002)Hans Bosch (TMI 2002)Knowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006

Variation 1 Variation 3Variation 2


2D + time matching2D + time matching


2D+time2D+time

van van der Geestder Geest (JCMR 2004)(JCMR 2004)


MultiMulti--view analysisview analysis

• Why?• Different views of same organ are correlated• Different views are complementary

• Goal:• Exploit coherence and redundancy

•• Why?Why?•• Different views of same organ are correlatedDifferent views of same organ are correlated•• Different views are complementaryDifferent views are complementary

•• Goal:Goal:•• Exploit coherence and redundancyExploit coherence and redundancy


MultiMulti--view AAM: different geometries view AAM: different geometries /phases/phases


MultiMulti--view AAM: Matchingview AAM: Matching

• Minimize intensity error for all views:• shape and appearance are coupled • pose varies independently

•• Minimize intensity error for all views:Minimize intensity error for all views:•• shape and appearance are coupled shape and appearance are coupled •• pose varies independentlypose varies independently


Contour detection: multiContour detection: multi--view AAMview AAM

Mehmet UzumcuMehmet UzumcuElco OostElco Oost


• Other applications: LV Angio•• Other applications: LV Other applications: LV AngioAngio

Elco Oost Elco Oost (FIMH 2005)(FIMH 2005)Knowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006

• Extension to 3D requires• Point correspondence• 3D procrustes alignment• 3D shape modeling• 3D matching mechanism

•• Extension to 3D requiresExtension to 3D requires•• Point correspondencePoint correspondence•• 3D 3D procrustes procrustes alignmentalignment•• 3D shape modeling3D shape modeling•• 3D matching mechanism3D matching mechanism


3D point correspondence3D point correspondence• Problem: no natural ordering in 3D

• Solutions:• Application specific manual (Mitchell,

Stegman)• Through parameterization

• Map to sphere– SPHARM (Brechbuhler)– MDL in 3D (Davies)

• M-reps (Pizer group)– Object coordinate frame

• Through registration • Deformable-mesh-to-binary-volume fitting (Kaus)• Non-rigid binary-binary registration (Frangi)

•• Problem: no natural ordering in 3DProblem: no natural ordering in 3D

•• Solutions:Solutions:•• Application specific manual (Mitchell, Application specific manual (Mitchell,

StegmanStegman))•• Through parameterizationThrough parameterization

•• Map to sphereMap to sphere– SPHARM (Brechbuhler)– MDL in 3D (Davies)

•• MM--reps (reps (Pizer Pizer group)group)– Object coordinate frame

•• Through registration Through registration •• DeformableDeformable--meshmesh--toto--binarybinary--volume fitting (volume fitting (KausKaus))•• NonNon--rigid binaryrigid binary--binary registration (binary registration (FrangiFrangi))


Application specific: manualApplication specific: manual• Interpolate in through plane direction

• Fix orientation

• Radial sampling of each contour

• In cardiac MR case: problem: slice shift!

•• Interpolate in through plane directionInterpolate in through plane direction

•• Fix orientationFix orientation

•• Radial sampling of each contourRadial sampling of each contour

•• In cardiac MR case: problem: slice shift!In cardiac MR case: problem: slice shift!


Eigen Eigen deformation #1deformation #1 EigenEigen deformation #2deformation #2

EigenEigen deformation #3deformation #3 EigenEigen deformation #4deformation #4

Van Van Assen Assen e.a.e.a.Knowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006

Point correspondencePoint correspondence• 3D Mesh-to-binary registration (Kaus)

• Represent one sample as triangle mesh• Rigidly align mesh to other samples• Non-rigid “snake-like” deformation, balancing

an external with an internal energy

+not restricted to spherical maps-non “optimal” parameterization

•• 3D Mesh3D Mesh--toto--binary registration (binary registration (KausKaus))•• Represent one sample as triangle meshRepresent one sample as triangle mesh•• Rigidly align mesh to other samplesRigidly align mesh to other samples•• NonNon--rigid “snakerigid “snake--like” deformation, balancing like” deformation, balancing

an external with an internal energyan external with an internal energy

+not restricted to spherical maps+not restricted to spherical maps--non “optimal” parameterizationnon “optimal” parameterization


Point correspondencePoint correspondence• Non-rigid registration (Frangi, Rueckert)•• NonNon--rigid registration (rigid registration (FrangiFrangi, , RueckertRueckert))


3D AAM Model Generation: MRI (Mitchell)

Average shape Tetrahedral representation


3D3D


4D AAM (4D AAM (StegmannStegmann))

c9285_opt_hires_c.avi


ASM MatchingASM Matching

• Updates on scanline can be generated by• Edge detector• Gray level model• Classifier

•• Updates on Updates on scanlinescanline can be generated bycan be generated by•• Edge detectorEdge detector•• Gray level modelGray level model•• ClassifierClassifier


ASM: MatchingASM: Matching

• Vector y = candidate points

• Vector = current model state• Align y to current shape in model frame• Scale y with • Update model parameters to match y

• Apply constraints on b• Per parameter:

• Mahalanobis distance:

• Repeat until convergence

•• Vector Vector yy = candidate points = candidate points

•• VectorVector = current model state= current model state•• Align Align yy to current shape in model frameto current shape in model frame•• Scale y with Scale y with •• Update model parameters to match yUpdate model parameters to match y

•• Apply constraints on Apply constraints on bb•• Per parameter:Per parameter:

•• MahalanobisMahalanobis distance: distance:

•• Repeat until convergenceRepeat until convergence

bxx Φ+=

)/(1 xy ⋅

)( xyb T −Φ=

iib λ3≤

t

t

i i

i Mb≤⎟⎟

⎠

⎞⎜⎜⎝

⎛∑=1

2

λ


ASM: matchingASM: matching

• Active Shape Models• Point distribution model with matching

•• Active Shape ModelsActive Shape Models•• Point distribution model with matchingPoint distribution model with matching

Movies: Tim Movies: Tim CootesCootes


3D ASM Matching: van 3D ASM Matching: van Assen Assen approachapproach

• Solution to voxel anisotropy• Piecewise 2D matching: update 3D model

state with 2D image info• Enables anisotropic data• Enables multiple orientations in data

•• Solution to Solution to voxelvoxel anisotropyanisotropy•• Piecewise 2D matching: update 3D model Piecewise 2D matching: update 3D model

state with 2D image infostate with 2D image info•• EnablesEnables anisotropicanisotropic datadata•• Enables multiple orientations in dataEnables multiple orientations in data


Iterative matching algorithmIterative matching algorithm

Next Next iterationiteration

Align model Align model to update to update

points cloud, points cloud, change bchange b--

vectorvector

Classify surrounding pixels, Classify surrounding pixels, and detect edge positionsand detect edge positions

Intersect with image planesIntersect with image planes Propagate updatesPropagate updates

3D model mesh3D model mesh

Place in data setPlace in data set

Map Map results results to meshto mesh


3D ASM on densely sampled data3D ASM on densely sampled data

Hans van Hans van AssenAssen


Application to sparse dataApplication to sparse data

Radial LA Multi-view (2SA+2LA) SARadial LARadial LA MultiMulti--view (2SA+2LA) view (2SA+2LA) SASA

Hans van Hans van AssenAssen, MEDIA 2006 , MEDIA 2006


3D ASM on sparse data: LA / SA fusion3D ASM on sparse data: LA / SA fusion



ASM versus AAMASM versus AAM

• 3 key differences:• ASM only uses texture in small region around

landmark, AAM complete patch• ASM searches around current positions, AAM

only under current position• ASM minimizes distance to boundary, AAM

intensity difference

•• 3 key differences:3 key differences:•• ASM only uses texture in small region around ASM only uses texture in small region around

landmark, AAM complete patchlandmark, AAM complete patch•• ASM searches around current positions, AAM ASM searches around current positions, AAM

only under current positiononly under current position•• ASM minimizes distance to boundary, AAM ASM minimizes distance to boundary, AAM

intensity differenceintensity difference


ASM versus AAM (ASM versus AAM (CootesCootes))

• General• ASM is faster• AAM possible with fewer landmarks• AAM has “interpretable” convergence criterion

• automatic failure detection (Thodberg, Stegmann)

•• GeneralGeneral•• ASM is fasterASM is faster•• AAM possible with fewer landmarksAAM possible with fewer landmarks•• AAM has “interpretable” convergence criterionAAM has “interpretable” convergence criterion

•• automatic failure detection (automatic failure detection (ThodbergThodberg, , StegmannStegmann))


Statistical model limitationsStatistical model limitations

• How to balance training set?

• How many samples necessary?

• Manual annotation of training samples

• Local refinement necessary

• Dimensionality vs nr training samples

•• How to balance training set?How to balance training set?

•• How many samples necessary?How many samples necessary?

•• Manual annotation of training samplesManual annotation of training samples

•• Local refinement necessaryLocal refinement necessary

•• Dimensionality Dimensionality vs vs nr training samplesnr training samples


3D ASM matching: 3D ASM matching: KausKaus approachapproach• Deformable model based on energy

• Shape model is integrated in Eint

• Term for PDM for each single surface• Term for connections between surfaces term• General shape & connectivity constraint term

• Local intensity model in Eext

• Every scanline has different feature detector (3 classes)

•• Deformable model based on energy Deformable model based on energy

•• Shape model is integrated in Shape model is integrated in EEintint

•• Term for PDM for each single surfaceTerm for PDM for each single surface•• Term for connections between surfaces termTerm for connections between surfaces term•• General shape & connectivity constraint termGeneral shape & connectivity constraint term

•• Local intensity model in Local intensity model in EEextext

•• Every Every scanline scanline has different feature detector has different feature detector (3 classes)(3 classes)

intEEE ext +=


Courtesy of Jens von Berg, Philips Research Laboratories, HamburCourtesy of Jens von Berg, Philips Research Laboratories, HamburggPresented at FIMH 2005Presented at FIMH 2005

Multi-surface Cardiac Modelling, Segmentation, and Tracking

Jens von Berg and Cris tian LorenzPhilips Res earch Labora tories , Hamburg














ComputerComputer--aided diagnosisaided diagnosis

• Construct a model of normal motion

• Classify and localize deviations from normal

• Demo

•• Construct a model of normal motionConstruct a model of normal motion

•• Classify and localize deviations from Classify and localize deviations from normalnormal

•• DemoDemo


Computer aided diagnosisComputer aided diagnosis

Avan Suinesiaputra Avan Suinesiaputra (MICCAI 2004)(MICCAI 2004)


Detection of Abnormal Contractility Detection of Abnormal Contractility PatternsPatterns

Avan SuinesiaputraAvan SuinesiaputraKnowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006

Case 1:Case 1:

Case 2:Case 2:

Avan SuinesiaputraAvan Suinesiaputra


RestRest--stress comparisonstress comparison

Avan Suinesiaputra Avan Suinesiaputra (IPMI 2005)(IPMI 2005)Knowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006














-=

),,(),,(11),,(zyxf

czyxfw

zyxb∇

−−=


Deformable thorax templateDeformable thorax template


= Left lung= Right lung= Air = Other



CMR scan planningCMR scan planning

CMR scan planningCMR scan planning

Scout views 2 Chamber view

4 Chamber view

Short-axis view




Manual

Automatic


Characteristic of Cardiac Respiratory Characteristic of Cardiac Respiratory Motion Motion

• Induced by changes in lung volumes

• Motion occurs in inferior and anterior direction. (truely 3D)

• Heart is adjacent to left lung

•• Induced by changes in lung volumesInduced by changes in lung volumes

•• Motion occurs in inferior and anterior Motion occurs in inferior and anterior direction. (truely 3D)direction. (truely 3D)

•• Heart is adjacent to left lungHeart is adjacent to left lung


Data fusion between scout & Data fusion between scout & perfusion scansperfusion scans



• Data from 3 infarct patients• Model fitted to fused feature points from

scouts / perfusion scan• Lung volume correlated between manual

contours and model lung

•• Data from 3 infarct patientsData from 3 infarct patients•• Model fitted to fused feature points from Model fitted to fused feature points from

scouts / perfusion scanscouts / perfusion scan•• Lung volume correlated between manual Lung volume correlated between manual

contours and model lungcontours and model lung














Right Ventricle Left Ventricle

A Virtual Exploring Robot for Left Ventricle Contour Detection

RV LV

RV LV


Sensor system (Perception devices)

Range SensorsNavigationContour Detection

Image Sensor (camera)RecognitionRegion Labeling

RV LV


Endocardium Detection

d

Contour point


Luca Luca FerrariniFerrarini, Hans , Hans OlofsenOlofsen,, Faiza BehloulFaiza Behloul


Epicardium delineation


Luca Luca FerrariniFerrarini














MA image processingMA image processing

Ernst Ernst Bovenkamp Bovenkamp (PATREC 2004)(PATREC 2004)


MA image processing: agent relationsMA image processing: agent relations

Ernst Ernst BovenkampBovenkampKnowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006

Example: conflict resolutionExample: conflict resolution

Ernst Ernst BovenkampBovenkamp


Example: testExample: test--hypothesishypothesis

Ernst Ernst BovenkampBovenkampKnowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006

Example: segmentation resultsExample: segmentation results














Flow

Data fusion: the challengeData fusion: the challenge

Function

PerfusionInfarct

Imaging

Rest Stress

Anatomy


Why Image (data) fusion ?Why Image (data) fusion ?

New featuresComplete “view”

Reduces uncertainty

Redundancy

Complementary


PET / MRI Symbolic data fusionPET / MRI Symbolic data fusion((BehloulBehloul e.a., TMI 2001)e.a., TMI 2001)

PET SPECT MRI EchoThe different

imaging Modalities

Dedicated feature extraction functions

Mapping functions

Abstract Description Template

Inference Level CAD System

LVRVAbstract LevelModality Independent

Application LevelModality dependent

PET SPECT MRI EchoPETPET SPECTSPECT MRIMRI EchoEchoThe different

imaging Modalities

Dedicated feature extraction functions

Mapping functions

Abstract Description Template

Inference Level CAD System

LVRVAbstract LevelModality Independent

Application LevelModality dependent


Data fusion: perfusion quantificationData fusion: perfusion quantification

M. Jerosch-Herold et al., JMRI, 2004


Data fusion: myocardial viabilityData fusion: myocardial viability

RestRest Stress

(low dose dobutamine)

Stress

(low dose dobutamine)


Rob van Rob van der Geestder Geest


Delayed enhancementDelayed enhancement Signal intensity

Transmural extent



Rob van Rob van der Geestder GeestKnowledge driven segmentation of cardiovascular images, UCLA IPAM, 7-2-2006

To summarize ….To summarize ….


• statistical shape and template models for segmentation, diagnosis

• Data driven• High level reasoning

• coordinating multiple segmentation sources• fusion of complementary information


•• statistical shape and template models for statistical shape and template models for segmentation, diagnosissegmentation, diagnosis

•• Data drivenData driven•• High level reasoningHigh level reasoning

•• coordinating multiple segmentation sourcescoordinating multiple segmentation sources•• fusion of complementary informationfusion of complementary information


What’s next?What’s next?

• Integrating the three main directions

• Automatic “quality control”

• Computer aided diagnosis

• Data fusion for segmentation and analysis

•• Integrating the three main directionsIntegrating the three main directions

•• Automatic “quality control”Automatic “quality control”

•• Computer aided diagnosisComputer aided diagnosis

•• Data fusion for segmentation and Data fusion for segmentation and analysisanalysis


AcknowledgementsAcknowledgementsRob van der Geest

Avan Suinesiaputra

Hans van Assen

Hans Bosch

Faiza Behloul

Luca Ferrarini

Maribel Adame

Elco Oost

Julien Milles

Steve Mitchell

Mehmet Uzumcu

Ernst Bovenkamp

Alejandro Frangi

Hans Reiber

Milan Sonka


Avan SuinesiaputraAvan Suinesiaputra


Hans BoschHans Bosch

Faiza BehloulFaiza Behloul

Luca Luca FerrariniFerrarini

Maribel AdameMaribel Adame

Elco OostElco Oost

Julien MillesJulien Milles

Steve MitchellSteve Mitchell

Mehmet UzumcuMehmet Uzumcu

Ernst Ernst BovenkampBovenkamp

Alejandro Alejandro FrangiFrangi

Hans Hans ReiberReiber

Milan Milan SonkaSonka

knowledge driven segmentation introduction of cardiovascular...

Documents