panel: image interpretation · 2010. 4. 15. · panel: image interpretation challenges and...

Panel: Image Interpretation

Challenges and Approaches to Standardization

Lawrence Schwartz, MDDepartment of Radiology

Columbia Presbyterian Medical [email protected]

Imaging Committee Chair for CALGB Wright Center of Innovation

in Biomedical Imaging

Imaging Modalities Used in Clinical TrialsImage Interpretation Standardization

CT

o The need for standardization varies by imaging modality, technique and potentially therapeutic optionMRI

PET

therapeutic option

o The need and degree of standardization is clearly related to the magnitude of the therapeutic effect which is to be measured

The Need for Interpretation StandardizationCT in Colorectal Cancer

post-Therapypre-Therapy

The Need for Interpretation StandardizationCT in Lung Cancer

Baseline Week 10 Month 10

The Need for Interpretation StandardizationPET in Lymphoma Cancer

The Need for Interpretation StandardizationWhat are sources of variability ?

o Target lesion selection

o Image acquisition protocols

o Measurement of target lesions

Interpretation of “clear unequivocal progression o Interpretation of “clear unequivocal progression of non-target disease”

o Identification of new lesions

o Primary tumor type

Categories of Lesions in RECIST

o Target

o Non Target

o New Lesion

Overall responses for all possible combinationsTarget lesions

Nontargetlesions

Newlesions

Overallresponse

CR CR No CR

CR Incomplete response/SD No PR

Table of Response AssessmentRECIST

CR Incomplete response/SD No PR

PR Non-PD No PR

SD Non-PD No SD

PD Any Yes or No PD

Any PD Yes or No PD

Any Any Yes PD

Response assessment Response Rank

Patient

No.

Total No .

of lesions

No. of

groupingsCR PR SD PD 1 2 3

No. of response

categories

1 7 21 0 0 0 21 1 0 0 1

2 16 4368 0 0 3697 671 0.85 0.15 0 2

3 10 252 0 100 152 0 0.6 0.4 0 2

4 10 252 1 232 19 0 0.98 0.08 0.004 3

The Need for Interpretation Standardization Variability - Target Lesion Selection

4 10 252 1 232 19 0 0.98 0.08 0.004 3

5 12 792 0 0 31 761 0.96 0.04 0 2

6 15 3003 0 0 3003 0 1 0 0 1

Calculated tumor response assessments, response ranks, and response

categories for one patient, analyzing 10 lesions with RECIST criteria

Clin Cancer Res. 2003 Oct 1;9(12):4318-23

STANDARIZED AVERAGE

RESPONSE VARIANCE

36 36 36 33 24 22 19 18 16 10 9 6

0.0 0.2 0.4 0.6 0.8 1.0

The Need for Interpretation Standardization Target Lesion Selection

NUMBERS OF LESIONS

PER GROUP

STANDARIZED AVERAGE

RESPONSE VARIANCE

1 2 3 4 5 6 7 8 9 10 11 12

0.0 0.2 0.4 0.6 0.8 1.0

Clin Cancer Res. 2003 Oct 1;9(12):4318-23

The Need for Interpretation Standardization Image Acquisition - Contrast Administration

Larry Schwartz

Memorial Sloan Kettering Cancer Center

The Need for Interpretation StandardizationCT Contrast Administration


Response = PR


Response = PD

Pre-walking CT Post-walking CT

Sources of VariabilityModality Acquisition and Measurement of target lesions

1.25-mm 1.25-mm

Uni-dimension (mm):

Bi-dimension (mm2):

Volume (mm3):

Pre-walking Post-walking Variation

27.6 27.8 0.7%

552 597.7 7.9%

4957.1 4852.3 2.1% Zhao Radiology

Concordance

correlation coefficient

Mean

% relative difference

95% Limits of

agreement� 95% CI

Sources of VariabilityModality Acquisition and Measurement of target lesions

Uni-

dimensional1.00 (1.00, 1.00) -0.6% -7.3 %, 6.2 %

Bi-

dimensional1.00 (0.99, 1.00) 1.1% -17.6 %, 19.8 %

Volume 1.00 (1.00, 1.00) 0.7% -12.1%, 13.4 %

Zhao Radiology

Sources of VariabilityInterpretation of “clear unequivocal progression of non-target

disease”

o There is no clear definition or interpretation of “clear unequivocal progression of non-target disease” in RECISTdisease” in RECIST

o This may result in variable interpretations impacting TTP image analysis especially in diseases with more extensive non target component

Sources of VariabilityInterpretation of “clear unequivocal progression of non-

target disease”

Sources of VariabilityIdentification of new lesions

Frequency of pulmonary nodules detectionNo. of Nodules Observer A Observer B

1.25 mm 5 mm 1.25 mm 5 mm

2-5 mm 28 13 36 22

Impact on lung lesion detection for time to progression analysis

2-5 mm 28 13 36 22

6-10 mm 18 14 20 18

11-30 mm 9 9 9 9

Total 55 36 65 49

What is an “optimal” or “acceptable”Agreement among observers?

o Consideration of the Primary Tumor and type of metastatic disease

o Mesothelioma

o Ovariano Colorectal

Renalo Ovarian

o Pancreas

o Gastric

o Others – Prostate, lymphoma

A single, standard agreement/adjudication rate would not reflect the

variability in assessments across clinical trials

o Renal

o Breast

Acceptable Adjudication Rate?Number of Modalities Assessed

o Case Study 1

o Nonsmall Cell Lung Cancer

CT – Chest

o Case Study 2

o Ovarian Cancer

o CT – Chest / Abdomen / Pelviso CT – Chest

/Abdomen

Abdomen / Pelvis

o FDG-PET

o CA-125

o QOL assessment

o Paracentesis for ascites

A single, standard agreement/adjudication rate would not reflect the

variability in assessments across clinical trials

Acceptable Agreement Rate?Each Adjudicated Case may not be Equal

SD SD PR PR PR PD

0 1 2 3 4 5 6 7 8months

Reader 1PR PR

SD SD PR PD

monthsReader 2

PDPD PDPD

Case 1

0 1 2 3 4 5 6 7 8months

SD SD PR PR PR PD

0 1 2 3 4 5 6 7 8months

Reader 1PR PR

SD SD PR PD

0 1 2 3 4 5 6 7 8months

Reader 2PR PR PR PD

Case 2

Waterfall Plot / AnalysisMay mandate even greater agreement….

0

0.2

-1

-0.8

-0.6

-0.4

-0.2

CALGB

o Founded 1956

o The unit of membership is the

US Cooperative Groups

o Cooperative groups are consortia of institutions that o The unit of membership is the

institution; 28 main members, 14 CCOPs, 225 affiliates

o (Headquarters): University of Chicago; Statistical Center: Duke University

institutions that conduct research in cancer treatment, prevention, biology and health outcomes

Treatment (Intervention) Trials @ CALGB

o Phase I or limited access n = 3

o Phase II n = 22

o Phase III n = 18

o Breast

o Lymphoma

o GI

o Colorectal, esophagus, o Phase III n = 18

o Registration directed (prospective) n = 4

o Several retrospective registration directed trials

o Colorectal, esophagus, rectal

o GU

o Kidney, bladder, prostate

o Pathology

o Imaging

Setting Standards of Care

o FDA approvals based on cooperative group data:

-cisplatin for NSCLC

-paclitaxel for ovarian and NSCLC

-paclitaxel as adjuvant therapy for breast cancer

-tamoxifen for breast cancer prevention-tamoxifen for breast cancer prevention

-interferon for high risk melanoma

-5-azacytidine for MDS

-oxaliplatin for met. CRC

-bevacizumab in 2nd line therapy for mCRC

New CALGB Trials Utilizing Imaging

Protocol Study Chair Imaging Co-Chair

CALGB40502 Hope Rugo, M.D. Deanna L. Kroetz, Ph.D.

CALGB40503 Maura Dickler, M.D. Federico Innocenti, M.D.

CALGB50303Wyndham H. Wilson, M.D., Ph.D. Andrew D. Zelentz, M.D., Ph.D.

Heiko Schoder, M.D.

CALGB50701 Barbara Grant, M.D. Lale Kostakoglu, M.D.

CALGB80302 David H. Ilson, M.D., Ph.D. Nathan Hall, M.D., Ph.D.CALGB80302 David H. Ilson, M.D., Ph.D. Nathan Hall, M.D., Ph.D.

CALGB140503 Nasser Altorki, M.D. Ernest Scalzetti, M.D.

CALGB80802 Ghassan Abou-Alfa, M.D. Lawrence Schwartz, M.D.

SWOG0816 Oliver W. Press, M.D., Ph.D., Heiko Schoder, M.D.

CALGB30803 Sarita Dubey, M.D. Ernest Scalzetti, M.D

CALGB50604 David J. Straus, M.D. Lale Kostakoglu, M.D.

CALGB50801 Ann S. LaCasce, M.D. Lale Kostakoglu, M.D.

CALGB30901 Arkadiusz Z. Dudek M.D., Ph.D. Ernest Scalzetti, M.D

CALGB50602 Sonali M. Smith, M.D. Heiko Schoder, M.D.

CALGB50201 Thomas Shea, M.D. Lawrence Schwartz, M.D.

CALGB50203 David J. Straus, M.D. Malik Juweid, M.D.

CALGB50404 Barbara Grant, M.D. Malik Juweid, M.D.

St u dy N u m be r St u dy Na m e p ts a c c u ra l To t a l St u d i e sRe ce iv e dC A L G B 1 4 0 5 0 3 A P ha s e I I I Ra n do m iz e d Tr ia l o f Lo b e c to m y v e r s u s S u b l o ba r R e s e c t io n fo rS ma l l (< /= 2 c m ) P e r ip h e r a l N o n- s ma l l C e l l L u n g Ca n c e r 9 9 2 4 1C A L G B 8 0 3 0 2 A P ha s e I I Tr ia l o f P r eo p e r a t i v e I r i n o t e ca n, C i sp la t i n a n d Ra d ia t io n i nE so p ha g ea l Ca n c e r 4 5 1 3 7C A L G B 5 0 7 0 1 A P ha s e I I Tr ia l o f Ex t e n d e d I n d u c t io n Ep r a t uz u ma b ( A n t i- C D 2 2M o n o c l o na l A n t i bo d y ) ( C A L G B I N D # 1 0 1 2 4 1 ) P l u s R i t u x i ma b i nP r e v io u s l y U n t r ea t e d Fo l l i c u la r N o n- H o d g k i n ' s L y m p h o ma ( N H L ) 6 1 1 0 4C A L G B 5 0 6 0 2 A P ha s e I I S t u d y o f Ga l i x i ma b ( A n t i- C D 8 0 ) fo r Pa t i e n t s w i t hR e la p s e d / R e fr a c to r y H o d g k i n L y m p h o ma 1 4 2 5C A L G B 5 0 3 0 3 P ha s e I I I Ra n do m iz e d S t u d y o f R- C H O P v. Do s e- A d j u s t e d E P O C H- R w i t hM o l e c u la r P r o f i l i n g i n U n t r ea t e d D e N o vo D i f f u s e La r g e B- C e l lL y m p h o ma s 5 3 1 5 1C A L G B 5 0 2 0 3 P ha s e I I Tr ia l o f Do x o r u b i c i n, V i n b la s t i n e a n d G e m c i ta b i n e ( A V G )C h e m o t h e r a p y fo r N o n- B u l k y S ta g e I a n d I I H o d g k i n L y m p h o ma 1 0 5 4 0 9C A L G B 4 0 5 0 3 E n do c r i n e T h e r a p y i n Co m b i na t io n w i t h A n t i- V E G F T h e r a p y : ARa n do m iz e d, Do u b l e- B l i n d, P la c e bo - Co n t r o l l e d P ha s e I I I Tr ia l o fE n do c r i n e T h e r a p y A l o n e o r E n do c r i n e T h e r a p y P l u s B e va c iz u ma b( N S C 7 0 4 8 6 5 : I N D 7 9 2 1 ) fo r Wo m e n w i t h H o r m o n e R e c ep to rPo s i t v e A d va n c e d B r ea s t Ca n c e r 5 7 1 3 3

C A L G B 4 0 5 0 2 A Ra n do m iz e d P ha s e I I I Tr ia l o f W e e k l y Pa c l i ta x e l Co m p a r e d to W e e k l yNa n o p a r t i c l e A l b u m i n Bo u n d N A B- Pa c l i ta x e l o r I x a b ep e l o n eCo m b i n e d w i t h B e va c iz u ma b a s F i r s t- L i n e T h e r a p y fo r Lo ca l l yR e c u r r e n t o r M e ta s ta t i c B r ea s t Ca n c e r 1 1 2 2 8 5C A L G B 5 0 2 0 1 A P ha s e I I S t u d y to E va l u ta e t h e Sa f e t y a n d E f f i ca c y o f Z e va l i n ( I N D # B BI N D 1 1 0 2 3 ) T h e r a p u t i c R e g i m e n i n Pa t i e n t s w i t h Tr a n s fo r m e dC D 2 0+ B- c e l l N o n- H o d g k i n ' s L y m p h o ma 7 2 4S W O G 0 8 1 6 A P H A S E I I T R I A L O F R E S P O N S E- A D A P T E D T H E R A P Y O F S T A G E I I I- I VH O D G K I N L Y M P H O M A U S I N G E A R L Y I N T E R I M P D G- P E T I M A G I N G 3 0 4 2

ICL

VT

NH

MA

•10.

•30.

•105.

•14.

•26.

•21.

•39.

•40.

•53.

�ICL – Collaborative Sites [106] at USA

•34.

•12.

•46.

•66.

•70.

•76.

•103.

•31.

•37.

•45.

•47.

•69.

•77.

•79.

•33.

•36.

•48.

•52.

•57. •38.

•49.

•1.

•7.

•16.

•22.

•25.

•28.

•44.

•82.

•98.

•4.

•9.

•64.

WA

OR

ID

MT

CAWY

ND

SD

MN

MINY

MEVT

NH

MA

RI

WI•52.

•54.

•85.

•99.

•58.

•68.

•71.

•90.

•75.

•92. •89.

CALGB Imaging Core Lab Overview Procedures and Services

Jun Zhang, PhD; Nathan C. Hall, MD, PhD; Michael V. Knopp, MD, PhD

The Ohio State University, Columbus

MA

CT

RI

NJ

DE

MD

DC

•11.

•19.

•26.

•42.

•60.

•80.

•32.

•91.

•104.

•34.

•51.

•62.

•84.

•2.

•35.

•55.

•101.

•23.

•43.

•50.

•86.

•8.

•6.

•30.

•96.

•13.

•15.

•17.

•18.

•20.

•46.

•56.

•61.

•81.

•88.

•106

CA

NV

UT

AZNM

CO

NE

KS

OK

TXLA

AR

MO

IA

INILWV

KY

TN

MSAL

FL

GA

SC

NC

VA

PAOH

•41.

•24.

•59.

•63.

•65.

•72.

•74.

•87.

•27.

CTRI

DE

MD

•5.

•29.

DC

NJ

•67.

•102

•73

•83.

•95.

•78.

Imaging Core Service

Clinical Trials Quality Control

Infrastructure Administrative

ICR

•Imaging Core Facilities

•Vendor Imaging Systems

•Vendor Workstations

•Dedicated Workstations

•Director

•Project Leader

•Project Manager

•Dedicated Individuals

SOP Audit

ICR

•Data receipt confirmation

•Data quality check report

•DCIOM De-identification

•ICR database

•Site education/training/approval

•Overall communication

•Regular trial report

•Lab meetings

•Training sessions

•Site credentialing

•Compliance monitoring

•Protocol Amendment

•Site Technical Manual

•Trial E-mail

•Web/FTP transfer

•Data management

•Post-processing

•Central review

•WebEx

•De-identification

•Equipment validation

ICL

Communication

Data

Submission

CD

FTP

Imaging

Complete

Sites

Discussion

Y

N

PhoneEmail WebEx

ICL

Committee

Decision

Bland

Review

Review Panel

+Problem ?

Quality Control Workflow in Clinical

Cancer Trials

Data

Management

Quality

Check

Clinical QC

DICOM

QC

Protocol

Compliance

Receipt

Confirmatio

n

Imaging Core Laboratory

Endpoint

Analysis

Regular

Review &

Report

QC Report

Storage

Archiving

Database

Database

N

≤ 24 hours ≤ 72 hours

De-

Identification

Y

Consult

Committee

Database

Update

Notify Site of

Decision

Compliant ?

Semi-automatic PET/CT Image QC Program

o The need for standardization varies by imaging modality and potentially therapeutic option


Centralized Data with Remote Review

• Vendor Advanced Workstation based

• Extended Brilliance Workspace

• Multi-Modality Workplace

Centralized Data Review• Centralized Data Review

• Data in one system

• Multiple reviewers

• Easy and Real-Time Access – Internet

PET-response (≥ 35%

SUV): 5-FU and Oxaliplatin

with concurrent RT

FDG PET/CT after induction chemo can identify

patients who benefit from changing chemo resulting in

improved response rates and PFS

Imaging Adaptive Trials

T3/4 or N1

Esophageal

Cancer

Pre-tx PET/CT

then Induction

Chemotherapy

with concurrent RT

PET day 22-

28 to

evaluate

response

Referral for

surgical resection

6 weeks post-RT

PET- no response (< 35%

SUV): CPT-11, Taxotere with

concurrent RTIntenseIntense need for real-time standardization ofAcquisition and INTERPRETATION

Real-time Adaptive Trial Support -

1. New studies received? - Monitor trial Email and Workstation for the Review

2. New Pt registration? – Monitor trial email and remind sites of data submission

3. Data Receipt Confirmation within 24 hours upon data receipt

4. Quality Check Report notification within 48 hours for ‘baseline’ and ‘final’, 24 hours for ‘interim’

5. For ‘non-compliant’ studies, contact imaging committee for a final decision.5. For ‘non-compliant’ studies, contact imaging committee for a final decision.

6. DICOM image De-identification

7. Remote Review Scheduling with Central Readers

8. Prepare the review form for readers

9. Real-time Data Review with reader(s)

10. Request for review results from readers

11. Notification of central review results to sites and Central Office

ACADEMIC EXPERT PANEL

REVIEW

72 HOUR TURN AROUND FROM

ACQUISITION TO INTERPRETATION

Panel: Image InterpretationChallenges and Approaches to Standardization

o Interpretation by its nature is both quantitative as well qualitative

o Critical is standardization of acquisition, analysis and results reporting

o Expert interpretationo Expert interpretation

o Training, education, experience – imaging and therapeutic specific

o The need for standardization varies by imaging modality and potentially therapeutic option


panel: image interpretation · 2010. 4. 15. · panel: image interpretation challenges and...

Documents