slide - 1 confidential. slide - 2 confidential slide - 3 confidential interalgorithm study using ct...
TRANSCRIPT
Slide - 1 Confidential
Slide - 2 Confidential
Slide - 3 Confidential
Interalgorithm Study using CT Images of synthetic
nodules……….
Slide - 4 Confidential
An inter-algorithm study, in the same way QIBA has been working on inter-reader,inter-scanner, and inter-site. We will also connect it to the analysis section of QIBA Profile.
The aim of the study is to estimate inter- and intra-algorithm variability by the volume estimation of synthetic nodules from CT scans of an anthropomorphic phantom (according to the work of the QIBA 1A Group (see Dr. Petrick‘s paper , SPIE 2011)
For the impelmentation of the obejctives a challenge could be organized (?)
Objectives of the QIBA 3A Group
Slide - 5 Confidential
Motivation for the Study
Study Design
Dataset
Algorithms
Analysis Protocol
Analysis method
Result evaluation
Result presentation
Result evaluation for the QIBA protocol
Study Motivation
Slide - 6 Confidential
Motivation
Slide - 7 Confidential
Changes in nodules volume is important for diagnosis, therapy planning, therapy response evaluation
Measuring volume changes requires high accuracy in measurement of absolute volume
Ground true has to be exact measured. This is not the case by data annotation (inter- and intra-observer variability)
Volumes of synthetic nodules are measured (high accuracy)
Therefore it make sense to use such data (as ground truth) in order to calculate accuracy measurement of algorithms
The study results could be combined with the QIBA 1A Group work. This combination will improve the QIBA volumetric CT Profile development.
The study has to be complementary to the study of the 1A Group:
1A study: Radiologists and synthetic nodules (inter- intra-observer
variability)
3A Study: Algorithms and synthetic nodules: intra-algorithms variability)
Using of the same technological basis (scanner type, Philips (1A Group data, synthetic nodules)) to secure independence for inter-scanner variability (except BIAS between different serial numbers)
Motivation
Slide - 8 Confidential
Describe the overall analysis procedure (use as example the Bio-change Challenge Protocol http://www.nist.gov/itl/iad/dmg/biochangechallenge.cfm: How to Participate in Biochange Challenge (Dr Fenimore)
Download and read the Bio-change Challenge Protocol and email or fax Statement of Interest to NIST.
Download the Bio-changeChallengeSeries from the NIST FTP site and from the NBIA RIDER collection as described in the Protocol.
Run your change analysis algorithm or CAD tool in your lab on the validation data.
Report your change results in one of the required formats and send a Participation Agreement signed by the your team leader to NIST by January 18, 2011.
NIST will analyze the reported results, comparing them to the limited available ground truth as described in the Protocol. NIST will provide Participants with individual analysis of their results. We will publish the results of the evaluation, without publicly identifying individual scores by Participant.
Analysis procedure (analysis Protocol according Bio-change or Volcano Challenge?)
Slide - 9 Confidential
Has to be discussed during the meeting March 17th 2011
The aim of the study is not a challenge to know:
Who is the best image analysis algorithm
The aim of the study is to gain knowledge for the QIBA profile (paragraph 9 concerns image analysis)
Avoid competition
Support Cooperation, conjoint approach
OUR CONTEXT is QIBA
Slide - 10 Confidential
Phantom data as used for the 1A Group Study (see Dr. Patrick’s publication, SPIE 2011)
1C clinical data (Dr Fenimore/ Phantom –synthetic data)
Training dataset
Test dataset
Or only test dataset (analyze without trainings dataset)
data pool specification (simple geometries, up to anatomical representations)
(if clear than we could propose an experimental design)
Nodule classification concerning position. Shape and margins of the nodule … (according to the QIBA Profile)
Algorithm classification
Analysis protocol:
each algorithm is applied to the dataset separately
each algorithms is applied to the “training” data set
each algorithms is applied to the “test” data set
Statistical analysis of the analysis results of each algorithm separately
Data/Nodules/Algorithms
Slide - 11 Confidential
Academia and non profit organizations
Industrial vendors
(for example possible vendors, according to the Volcano 2009 challenge could be: Siemens, Phipps, MeVis, Kitware, Definiens, VIA CAD etc…
Description/Classification of the algorithms: according to the grade of user intervention is needed (for example Volcano’09, A. P. Reeves et al):
Totally automatic using seed points
Limited parameter adjustment (on less than 15% of the cases)
Moderate parameter adjustment (on less than 50% of the cases)
Extensive parameter adjustment (more than 50% of the cases)
Limited image/boundary modification (on less than 15% of the cases)
Moderate image/boundary modification (on less than 50% of the cases)
Extensive image/boundary modification (more than 15% of the cases)
Or:
M. Gavrielide et al., Noncalcified Lung Nodules, Volumetric assessment with thoracic CT (RSNA 2009):
Semi automated
Manually derived
Semi automated algorithms are typically initiated by defining a region of interest around a nodule or by a user-provided point inside
the nodule area. Depending on the application, segmentation algorithms are then employed to delineate nodules from the surrounding lung parenchyma and neighboring structures such as attached vasculature and pleural surfaces.
Manually derived methods require users to interactively delineate nodule boundaries, typically in a section-by-section fashion; this is followed by an application of three-dimensional software to merge the two-dimensional boundaries into a volume. The estimate of nodule volume is then based on the total number of voxels within the segmented region.
The majority of volume measurement methods use voxel counting
Algorithm vendors and Algorithm classification examples from the literature
Slide - 12 Confidential
IEEE Transactions on medical imaging, vol. 28 MICCAI
Algorithm Classification:
An automatic segmentation algorithm does not require any user intervention.
A semi- automatic algorithm needs minimal amount of input from user, e.g., a seed point to initialize the segmentation.
An interactive algorithm requires manual editing of the final results.
Abstract: In this paper, we present the organization of a competition of liver tumor segmentation techniques. The liver tumor segmentation competition is part of the workshop "3D Segmentation in the Clinic:
A Grand Challenge II" at Medical Image Computing and Computer Assisted Intervention 2008 conference. The goal of this contest is to compare the he performance of different algorithms for segmenting liver tumor from contrast enhanced CT images. Several organizing topics are described, contrast including the motivation for organizing this competition, training and testing dataset evaluation methods
Algorithm Classification according 3D Segmentation in the Clinic:
A Grand Challenge II - Liver Tumor Segmentation, Xiang Deng and Guangwei Du, Center for Medical Imaging Validation, Siemens Corporate Technology
(citation)
Slide - 13 Confidential
MICCAI Liver Challenge proposed metrics (citation)
Slide - 14 Confidential
Slide - 15 Confidential
Sensitivity
Sens = TP/TP+FN
Specificity
Spec = TN/TN+FP
Accuracy = (TN +TP)/all
Lucila Ohno-Machado
An introduction to calibration and discrimination methods, HST951 Medical Decision SupportHarvard Medical School, Massachusetts Institute of Technology
Slide - 16 Confidential
Box plot (for example: for each data set, nodule volume vs. method, relative Volume)
Volrel= 100%* (Volest – Voltrue)/ Voltrue
Volrel= 100%* (Volalg – Volsynthetic nodule)/ Volsynthetic nodule
We propose: A simple but well defined and reliable approach
Other
Slide - 17 Confidential
We propose: keep it simple
Algorithm Classification:
•An automatic segmentation algorithm does not require any user intervention.
•A semi- automatic algorithm needs minimal amount of input from user, e.g., a seed point to initialize the segmentation.
•An interactive algorithm requires manual editing of the final results.
Slide - 18 Confidential
Known: Vsn Measurement: Valg
StdvValg (1) <- same Phantom, multiple measurements, same algorithm, multiple seed points (two groups of algorithms: semiautomatic, manually)
StdvValg (2)<- different phantoms (according QIBA profile: nodule classification), same algorithm: • StdValg (nodule class)• StdValg(all nodule classes)
Bias out of digitalization in the area of lesion margins (Voxels/Pixels on the lesion margins, mixed pixels) (Group 1A, Group 1C, Group 1B, QIBA VIA Profile, Definitions and how should be calculated?)
Achievements:
• Transparent and Objective comparability between different algorithms
• One independent Error value for further investigations of clinical data
Algorithm Evaluation Method
Slide - 19 Confidential
Definition of the Study Data/Measurements/Metrics /
Which data:
1A (Phantom/synthetic data) (defined experimental conditions, experimental design
Phantom Data for dV?)
1B Clinical data
according to that: do we have we to separate the study design as following?
Study design for Phantom/Synthetic data (focus)
Study design for Clinical data
Metrics
Slide - 20 Confidential
Decision
Slide - 21 Confidential
Each participant has to be informed (only their own algorithm results)
Publication of the results (all participants)
Using the results for the QIBA protocol: knowledge exploitation for the QIBA protocol
Resulting recommendations for FDA (Is this possible?)
Result information for the participants