the lung image database consortium (lidc) data collection process this presentation based on the...

The Lung Image Database Consortium (LIDC) The Lung Image Database Consortium (LIDC) Data Collection ProcessData Collection Process

This presentation based on the RSNA 2004 InfoRAD This presentation based on the RSNA 2004 InfoRAD theater presentation titledtheater presentation titled

“The Lung Imaging Database Consortium (LIDC) : “The Lung Imaging Database Consortium (LIDC) : Creating a Publicly Available Database to Stimulate Creating a Publicly Available Database to Stimulate

Research in CAD Methods for Lung Cancer”Research in CAD Methods for Lung Cancer”

(9110 DS-i)(9110 DS-i)November 29, 2004November 29, 2004

Michael McNitt-Gray (UCLA), Anthony P. Reeves (Cornell), Michael McNitt-Gray (UCLA), Anthony P. Reeves (Cornell),

Roger Engelmann (U. Chicago), Peyton Bland (U. Michigan), Roger Engelmann (U. Chicago), Peyton Bland (U. Michigan),

Chris Piker (U. Iowa), John Freymann (NCI) and Chris Piker (U. Iowa), John Freymann (NCI) and

The Lung Image Database Consortium (LIDC)The Lung Image Database Consortium (LIDC)

Principal Principal GoalsGoalsTo establish standard formats and To establish standard formats and

processes for managing thoracic CT scans processes for managing thoracic CT scans and related technical and clinical data for and related technical and clinical data for use in the development and testing of use in the development and testing of computer-aided diagnostic algorithms.computer-aided diagnostic algorithms.

Principal Principal GoalsGoalsTo establish standard formats and To establish standard formats and

processes for managing thoracic CT scans processes for managing thoracic CT scans and related technical and clinical data for and related technical and clinical data for use in the development and testing of use in the development and testing of computer-aided diagnostic algorithms.computer-aided diagnostic algorithms.

To develop an image database as a web-To develop an image database as a web-accessible international research resource accessible international research resource for the development, training, and for the development, training, and evaluation of computer-aided diagnostic evaluation of computer-aided diagnostic (CAD) methods for lung cancer detection and (CAD) methods for lung cancer detection and diagnosis using helical CT.diagnosis using helical CT.

The The DatabaseDatabase•The database will contain:The database will contain:

1)1) A collection of CT scan imagesA collection of CT scan images2)2) Technical factors about the CT Technical factors about the CT

scanscan• Non-patient information in DICOM headerNon-patient information in DICOM header

3)3) For Nodules > 3 mm diameterFor Nodules > 3 mm diameter• Radiologist drawn boundariesRadiologist drawn boundaries• Description of characteristicsDescription of characteristics

4)4) For Nodules < 3 mmFor Nodules < 3 mm• Radiologist marks centroid, no Radiologist marks centroid, no

characteristicscharacteristics

5)5) Pathology results or diagnosis Pathology results or diagnosis information whenever availableinformation whenever available

6)6) All in a searchable relational All in a searchable relational databasedatabase

How to do this?How to do this?The LIDC Data Collection ProcessThe LIDC Data Collection Process

For nodule detection, recent research For nodule detection, recent research has demonstrated that the results has demonstrated that the results from a single reader are not sufficientfrom a single reader are not sufficient

How to do this?How to do this?The LIDC Data Collection ProcessThe LIDC Data Collection Process

At least two and perhaps four readers At least two and perhaps four readers may be required.may be required.

Not practical to do joint reading Not practical to do joint reading sessions across five institutionssessions across five institutions

LIDC Will NOT do a forced consensus LIDC Will NOT do a forced consensus read. We won’t force agreement on read. We won’t force agreement on location of a nodule nor its boundary.location of a nodule nor its boundary.

Truth – DetectionTruth – DetectionLIDC – Initial ApproachLIDC – Initial Approach

Multiple Reads with Multiple ReadersMultiple Reads with Multiple Readers First Read – 4 readers, each reads First Read – 4 readers, each reads

independently (Blinded)independently (Blinded) Compile 4 blinded reads and distribute to Compile 4 blinded reads and distribute to

readersreaders Second Read – Same 4 readers, this time Second Read – Same 4 readers, this time

unblinded to the results of the other unblinded to the results of the other readers from the first reading.readers from the first reading.

Still, no forced consensus on either Still, no forced consensus on either location of nodules nor on their location of nodules nor on their boundaries.boundaries.

Blinded Reads – Each Reader Reads Independently (Blinded to Results of Other Readers)

Reader 1

Blinded Read for Reader 1 – Marks Only One Nodule

Reader 2

Blinded Read for Reader 2 – Marks Two Nodules(Note: One nodule is same as Reader 1)

Reader 3

Blinded Read for Reader 3 – Marks Two Nodules(Note: Again, One nodule is same as for Reader 1)

Reader 4

Blinded Read for Reader 4 – Did Not Mark Any Nodules

2nd Round - UnBlinded Reads Readings in Which Readers Are Shown Results of Other Readers

Each Reader Marks Nodules After Being Shown Results From Their Own and Other Readers’ Blinded Reads (Each Reader Decides to Include or Ignore).

Reader 1

Unblinded Read for Reader 1 – Now Marks Two Nodules(Originally only marked one)

Reader 2

Unblinded Read for Reader 2 – Still Marks Two Nodules(No Change)

Reader 3

Unblinded Read for Reader 3 – Now Marks Three Nodules(Originally only marked two)

Reader 4

Unblinded Read for Reader 4 – Now Marks Three Nodules(Originally did not mark any)

4/4 Markings

2/4 Markings 2/4 Markings

Results of Unblinded Reads from All Four Readers

We will capture one aspect of reader variability in this way

Radiologist Review & Reconcile- V2

•4 radiologist (blinded) – R1B, R2B, R3B, R4B



•Submit to Requesting Site; This site compiles markings and re-sends case




•4 Radiologists see all (anonymized) markings

Radiologist Review & Reconcile

•4 Radiologists Perform Blinded Read – R1B, R2B, R3B, R4B


•4 Radiologists see all (anonymized) markings

•4 Radiologists Perform Unblinded Read (R1U, R2U, R3U, R4U)

Database (will contain Blinded AND

Unblinded reads)

R1U R2U R3U R4U

Nodules for each condition: (R1B, R2B, R3B, R4B, R1U, R2U, R3U, R4U)• Location • Outline (where appropriate)• Label (where appropriate)

Case 5, Slice 19Case 5, Slice 19

Radiologist 1 - Method Radiologist 1 - Method 11

For each voxel, sum the number of For each voxel, sum the number of occurrences (across reader markings) that it occurrences (across reader markings) that it was included as part of the nodulewas included as part of the nodule

Create a probabilistic map of nodule voxelsCreate a probabilistic map of nodule voxels Higher probability voxels are shown as Higher probability voxels are shown as

brighter; lower probability are darkerbrighter; lower probability are darker Can use apply a threshold and show only Can use apply a threshold and show only

voxels > some prob. Value if desired.voxels > some prob. Value if desired.

How to Represent This How to Represent This Variability? Create a Probabilistic Variability? Create a Probabilistic Description of Nodule BoundaryDescription of Nodule Boundary

Probabilistic Description of Probabilistic Description of BoundaryBoundary

Apply Threshold if Apply Threshold if DesiredDesired

Challenge: Define the Boundary of a Challenge: Define the Boundary of a NoduleNodule

Do we need to have agreement between Do we need to have agreement between radiologists on boundaries?radiologists on boundaries?

LIDC’s answer is no.LIDC’s answer is no.

LIDC Approach will be to:LIDC Approach will be to: Construct a probabilistic description of Construct a probabilistic description of

boundaries to capture reader variabilityboundaries to capture reader variability Use a threshold value (50% centile or 1% Use a threshold value (50% centile or 1%

centile) to give fixed contours.centile) to give fixed contours.

Pathology InformationPathology Information

In those cases in which pathology is In those cases in which pathology is available, we will extract from available, we will extract from reports:reports: Whether histology or cytology was Whether histology or cytology was

performedperformed If histology, try to establish the cell type If histology, try to establish the cell type

according to WHO classificationsaccording to WHO classifications If cytology, establish whether it was If cytology, establish whether it was

benign or malignantbenign or malignant

Pathology InformationPathology Information

If no pathology, other diagnostic If no pathology, other diagnostic information may be substituted when information may be substituted when available (such as 2 years Dx F/U with available (such as 2 years Dx F/U with no change in radiographic no change in radiographic appearance).appearance).

If neither is available, then case will If neither is available, then case will be used for detection purposes only.be used for detection purposes only.

Database ImplementationDatabase Implementation

How to capture and collect all of this data?How to capture and collect all of this data?5 Phases of Data Collection5 Phases of Data Collection

1.1. Initial ReviewInitial Review review case for inclusion in database; review case for inclusion in database; anonymize case; anonymize case; Index case, e.g. Full Chest/Limited Chest, Image Quality.Index case, e.g. Full Chest/Limited Chest, Image Quality.

2.2. Blinded ReadBlinded Read identifying and drawing nodules independentlyidentifying and drawing nodules independently

3.3. Unblinded ReadUnblinded Read confirming using an overread, labeling nodules confirming using an overread, labeling nodules

(characteristics)(characteristics)4.4. Subject infoSubject info

demographics, smoking history, pathology.demographics, smoking history, pathology.5.5. Export Data to NCI-hosted database (public)Export Data to NCI-hosted database (public)


How to capture and collect all of this data?How to capture and collect all of this data?

We have developed an internal standard for We have developed an internal standard for representing a representing a region of interest (ROI)region of interest (ROI) that that is 3-D based on xml. This is portable across is 3-D based on xml. This is portable across software drawing tools.software drawing tools.

We are also using xml to capture radiologist We are also using xml to capture radiologist interpretation of interpretation of nodule characteristicsnodule characteristics (shape, subtlety, etc.) by using a limited set (shape, subtlety, etc.) by using a limited set of descriptorsof descriptors


How to capture and collect all of this data?How to capture and collect all of this data?

We have designed and tested a We have designed and tested a communication protocolcommunication protocol to send image data to send image data and xml messagesand xml messages

Read Read RequestRequest messages (with a messages (with a code/mechanism to distinguish blinded from code/mechanism to distinguish blinded from unblinded read request)unblinded read request)

Read Read ResponseResponse messages (with a messages (with a code/mechanism to distinguish blinded from code/mechanism to distinguish blinded from unblinded read response)unblinded read response)


How to capture and collect all of this How to capture and collect all of this data?data?

Designed and implemented database Designed and implemented database for each host site for all case data.for each host site for all case data.

Designed and are implementing the Designed and are implementing the central NCI hosted database. central NCI hosted database.


Communication ModelCommunication Model Each Site Plays Dual RolesEach Site Plays Dual Roles

As a Requesting SiteAs a Requesting Site Identify Case and collect dataIdentify Case and collect data Phase 1- Initial ReviewPhase 1- Initial Review Manage it through blinded and unblinded read processManage it through blinded and unblinded read process Create database entry for caseCreate database entry for case Phase 4 – Demographics, PathologyPhase 4 – Demographics, Pathology Phase 5 – Export to NCIPhase 5 – Export to NCI NOTE: Site does not READ/MARK its own casesNOTE: Site does not READ/MARK its own cases

As a Servicing SiteAs a Servicing Site Perform blinded (Phase 2) and unblinded (Phase 3) Perform blinded (Phase 2) and unblinded (Phase 3)

readsreads

A

B,C,D,E

Initial Review,

Anonymize

Send Image Data

XML Reading Assignment message

XML Reading Response Message,

Compile Responses

Nodule Marking Tools

SSH or SCP for transfer

Other Subject data fieldsLinked to case

1

5

116

3

2

4

LIDC Message SystemX

7

9

8

10

Requesting Site

Servicing Site

Access to LIDC DatabaseAccess to LIDC Database Cases Exported to NCICases Exported to NCI NCI hosts DatabaseNCI hosts Database Publicly AvailablePublicly Available Query Based on Data Elements CollectedQuery Based on Data Elements Collected

Imaging Data such as Slice Thickness, etc.Imaging Data such as Slice Thickness, etc. Pathology or F/U DataPathology or F/U Data Other FieldsOther Fields

ObtainObtain Image Data including DICOM headersImage Data including DICOM headers Serial Imaging when availableSerial Imaging when available Radiologists’ Identification, Contours and Characterization of Radiologists’ Identification, Contours and Characterization of

NodulesNodules Diagnosis Data (Path, Radiographic F/U, etc) whenever availableDiagnosis Data (Path, Radiographic F/U, etc) whenever available Case Demographics whenever availableCase Demographics whenever available

Currently Implementing MIRC model (see infoRAD exhibit for Currently Implementing MIRC model (see infoRAD exhibit for demo)demo)

Database ImplementationDatabase ImplementationTASKS COMPLETED (see reports on TASKS COMPLETED (see reports on

website):website): Specification of Inclusion Criteria:Specification of Inclusion Criteria:

CT scanning technical parametersCT scanning technical parameters Patient inclusion criteriaPatient inclusion criteria

Process Model for Data collectionProcess Model for Data collection Determination of Spatial "truth" Using Blinded Determination of Spatial "truth" Using Blinded

and Unblinded readsand Unblinded reads Development of Boundary Drawing ToolsDevelopment of Boundary Drawing Tools Development and implementation of xml Development and implementation of xml

standard for ROIsstandard for ROIs


TASKS COMPLETEDTASKS COMPLETED Defined Common Data Elements for Defined Common Data Elements for

LIDCLIDC Database design – tables and Database design – tables and

relationships between tablesrelationships between tables Communication protocolCommunication protocol Establishing Public Database and Establishing Public Database and

Access Mechanism at NCIAccess Mechanism at NCI

Other Products Other Products Publications/PresentationsPublications/Presentations

LIDC Overview manuscriptLIDC Overview manuscript Radiology 2004 Sep;232(3):739-748. Radiology 2004 Sep;232(3):739-748.

Assessment Methodologies manuscriptAssessment Methodologies manuscript Academic Radiology April 2004 Academic Radiology April 2004

((Acad Radiol 2004; 11:462–475)

Special Session SPIE Medical Imaging Special Session SPIE Medical Imaging Sunday evening session at SPIE, 2005Sunday evening session at SPIE, 2005

SummarySummary

LIDC mission – to create public LIDC mission – to create public databasedatabase

Current understanding of problem dictated Current understanding of problem dictated multiple readersmultiple readers

Multi-Institutions dictated distributed, Multi-Institutions dictated distributed, asynchronous readsasynchronous reads

SummarySummary

LIDC developed:LIDC developed: Process Model for Blinded and Unblinded Reads Process Model for Blinded and Unblinded Reads

w/Multiple Readersw/Multiple Readers Infrastructure to Communicate Radiologist Expert Infrastructure to Communicate Radiologist Expert

InformationInformation (Markings, Contours, Labelings) (Markings, Contours, Labelings) Data Elements –image, meta data (DICOM), radiologist Data Elements –image, meta data (DICOM), radiologist

markings, contours and labels, pathology, demographicsmarkings, contours and labels, pathology, demographics Data Representation Scheme (xml)Data Representation Scheme (xml) Communication (messaging) protocolCommunication (messaging) protocol Database DesignDatabase Design Mechanism to handle reader disagreement/variabilityMechanism to handle reader disagreement/variability

the lung image database consortium (lidc) data collection process this presentation based on the...

Documents

available database

institutions lidc

blinded reads

sufficient slide

nodule detection

lung cancer detection

clinical data

recent research