![Page 1: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/1.jpg)
PET/CT Working Group Update
Jayashree Kalpathy-CramerSandy Napel
![Page 2: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/2.jpg)
JKC
Sub-group of the Image Analysis and Performance Metrics (IAPMWG) consisting of teams working in the areas of CT and PETRepresentation from
BWHColumbia UniversityIowaMGHMSKCCMoffittUPMCUWStanford
PET-CT Working Group
3/27/2014 PET-CT Working Group Update 2
![Page 3: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/3.jpg)
JKC
Multi-site algorithm comparison
Task: CT-based lung nodule segmentationEvaluate algorithm performance
Bias, repeatability of volumesOverlap measuresUnderstand sources of variability
CT Segmentation Challenge
3/27/2014 PET-CT Working Group Update 3
![Page 4: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/4.jpg)
JKC
CUMC: marker-controlled watershed and geometric active contours Moffitt Cancer Center: multiple seed points with region growing. Ensemble segmentation obtained from the multiple grown regions. Stanford University: 2.5 dimension region growing using adaptive thresholds initialized with statistics from a “seed circle” on a representative portion of the tumor
Participants and Algorithms
3/27/2014 PET-CT Working Group Update 4
![Page 5: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/5.jpg)
JKC
52 nodules from 5 collections hosted in The Cancer Imaging Archive (TCIA)
LIDC (10 studies with 1 nodule each)RIDER (10 studies with 1 nodule each)CUMC Phantom (single study, 12 nodules)Stanford (10 studies with 1 nodule each) Moffitt (10 studies with 1 nodule each)
Data
3/27/2014 PET-CT Working Group Update 5
![Page 6: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/6.jpg)
JKC
Distribution of volumes in collections
3/27/2014 PET-CT Working Group Update 6
Nodules in the LIDC and phantom collection were small while other collections had a wide
range of nodule sizes
![Page 7: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/7.jpg)
JKC
Created converters for a range of data formats (PNG, AIM, DICOM-SEG, DICOM-RT, .MAT, LIDC-XML)Used TaCTICS to compute metrics
C++ ITK libraries (20+ metrics)R statistics engine (statistical analysis and visualization)
Agreed to use DICOM-SEG or DICOM-RT for future segmentation challengesExploring use of NCIPHUB for future challenges
Informatics
3/27/2014 PET-CT Working Group Update 7
![Page 8: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/8.jpg)
JKC
Ground truth: volume of nodules in phantom known(Approximate truth): consensus segmentation obtained using submitted segmentations (STAPLE, thresholded probability map, majority vote)Each group submitted at least 3 results for each algorithmBias: estimate volume of algorithms compared to known truth (based on phantom data)Reproducibility: calculated using multiple segmentations submitted for each algorithm
Evaluation
3/27/2014 PET-CT Working Group Update 8
![Page 9: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/9.jpg)
JKC
Volume differences: based on number of voxels in each volumeDoes not take into account the spatial locations of the respective volumesNot symmetric
Volumetric difference
3/27/2014PET-CT Working Group Update, QIN F2F 2014
9
![Page 10: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/10.jpg)
JKC
Dice coefficient
3/27/2014 PET-CT Working Group Update 10
Dice (and Jaccard) coefficients most commonly used measures of spatial overlap for binary labels
symmetricover or under-segmentation errors are weighted equally
Spatial overlap measures depend on the size and shape of the object as well as the voxel size relative to the object size
![Page 11: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/11.jpg)
JKC
Hausdorff Distance
3/27/2014PET-CT Working Group Update, QIN F2F 2014
11
The Hausdorff Distance (HD) between A and G, h(A, G) is the maximum distance from any point in A to a point in G and is defined as
![Page 12: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/12.jpg)
JKC
Distribution of Dice coefficients
3/27/2014 PET-CT Working Group Update 12
Pairwise Dice coefficients were calculated between all segmentations for a given nodule
Intra-algorithm agreement was much higher than inter-algorithm agreement (p <0.05)
![Page 13: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/13.jpg)
JKC
Dice coefficients by collection
3/27/2014 PET-CT Working Group Update 13
All pairwise dice coefficients (all runs, all algorithms by nodule) by collection shows better agreement between algorithms on the phantom
nodules (CUMC) than on clinical data
![Page 14: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/14.jpg)
JKC
Dice coefficient (all algorithms, all runs) of
nodules in Stanford collection (ordered by volume left to right)
Exploring causes of variability
3/27/2014 PET-CT Working Group Update 14
Estimated volume varies significantly by algorithm
![Page 15: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/15.jpg)
JKC
Some nodules (e.g., Lg from the Stanford collection) have high
variability (typically heterogeneous)
Exploring causes of variability
3/27/2014 PET-CT Working Group Update 15
![Page 16: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/16.jpg)
JKC
Estimating Bias in phantom data
3/27/2014 PET-CT Working Group Update 16
Bias (estimated-true volume) for CUMC-phantom nodules shows a difference between algorithms
(ANOVA with blocking, p <<0.05)
![Page 17: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/17.jpg)
JKC
Patterns of bias are different in large vs. small nodules
Bias in small and large nodules
3/27/2014 PET-CT Working Group Update 17
![Page 18: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/18.jpg)
JKC
Algorithms are not perfectly deterministic (i.e different
segmentations yield different volumes)
Reproducibility of algorithms
3/27/2014 PET-CT Working Group Update 18
![Page 19: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/19.jpg)
JKC
Dice coefficients between segmentations generated by a given algorithm vary between algorithms
Reproducibility of algorithms
3/27/2014 PET-CT Working Group Update 19
![Page 20: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/20.jpg)
JKC
Catalog of CT segmentation toolsFeature extraction project: Assess impact of segmentations on features (shape, texture, intensity) implemented at different QIN sites
Comparison of features by implementationComparison by feature type
CT Segmentation: Future plans
3/27/2014 PET-CT Working Group Update 20
![Page 21: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/21.jpg)
JKC
Four (+?) phase challenge:software phantom (DRO)hardware phantom scanned at multiple sitessegmenting clinical data correlating PET with outcomesdynamic PET (MSKCC)
PET Segmentation Challenge
3/27/2014 PET-CT Working Group Update 21
![Page 22: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/22.jpg)
JKC
Generated by UW/QIBA7 QIN sites participatedUW, Moffitt, Iowa, Stanford, Pittsburgh, CUMC, MSKCCSoftware packages used included PMOD, Mirada Medical RTx, OSF tool, RT_Image, CuFusion, 3D Slicer, Osirix, AmideAfter some effort, all sites were able to calculate the DRO SUV metrics correctly
Digital Reference Object (DRO)
3/27/2014 PET-CT Working Group Update 22
![Page 23: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/23.jpg)
JKC
Use michallenges.org to distribute data and post challenge rulesExploring use of nciphub.org for challenges going forward
PET segmentation challenge
Informatics
3/27/2014 PET-CT Working Group Update 23
![Page 24: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/24.jpg)
JKC
Phase II: Hardware phantom scanned at 2+ sites (UI, UW)NEMA IEC Body Phantom Set™Model PET/IEC-BODY/P Four Image Sets per SiteGenerate accurate volumetric segmentations of the objects in the phantom scans
Hardware phantom
3/27/2014 PET-CT Working Group Update 24
Calculate the following indices for each of the objects: VOI volume, Max, PEAK & AVERAGE Concentration, Metabolic Tumor Volume
![Page 25: PET/CT Working Group Update Jayashree Kalpathy-Cramer Sandy Napel](https://reader036.vdocument.in/reader036/viewer/2022062515/56649cf85503460f949c92f2/html5/thumbnails/25.jpg)
JKC
LeadershipSandy Napel: WG chairKaren Kurdzeil: WG co-chair
MilestonesTool CatalogPET segmentation challengesCT feature extraction challenges
Future Plans
3/27/2014 PET-CT Working Group Update 25