![Page 1: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/1.jpg)
Comparing the accuracy of prediction methods
Michael W. Kattan, Ph.D.
Associate Attending Outcomes Research Scientist
Memorial Sloan-Kettering Cancer Center
Associate Professor of Public HealthCornell University
![Page 2: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/2.jpg)
How is risk typically computed?
• Based on features, we make a crude tree.
• Most cancer staging systems do this.
BT=high
H=AggAnd
DE=E
HIGH RISK LOW
RISK
YN
Y
N
![Page 3: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/3.jpg)
The problem with crude trees
• They are very easy to use.
• But they do not predict outcome optimally.» High risk groups are very heterogeneous.
– A single risk factor may qualify a patient as high risk.
• Other approaches, like a Cox regression model, predict more accurately.
![Page 4: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/4.jpg)
Some simple steps that will make a difference
1. Build the most accurate model possible.2. Take model to bedside
» As a nomogram, » In stand-alone software (desktop, handheld, web)» Built into the electronic medical record
• Doing this will predict patient outcome more accurately, resulting in» better patient counseling» better treatment decision making
![Page 5: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/5.jpg)
Desirable characteristics of an error measure
• Understandable/interpretable
• Sensitive to model improvement
• Model-free
• Unaffected by censoring
![Page 6: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/6.jpg)
CONCORDANCE INDEX (censored data)
• probability that, given two randomly drawn patients, the patient who fails first had a higher probability of failure.
• assumes that the patient with the shorter follow-up fails
• does not apply if both patients fail at the same time, or the censored patient has shorter follow-up.
Usable patient pairs with consistent outcomeUsable patient pairs
usable patient pair - patient with the shorter follow-up must fail
consistent outcome - failure more likely for the shorter follow-up patient
tied predicted probabilities get 1/2 (Harrell, 1982)
![Page 7: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/7.jpg)
Gastric Cancer Disease-Specific Survival by AJCC Stage
![Page 8: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/8.jpg)
Gastric Cancer Disease-Specific Survival Nomogram
MM
SM
MP
SS S2
S3S1
Kattan et al., JCO, 2003
![Page 9: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/9.jpg)
How to tell if we are doing any better than existing models?
Compare jackknife predicted probabilities of new model to existing model predictions:
Method Concordance Index
AJCC Stage 0.77
Nomogram (jackknife) 0.80
(p<0.001).
![Page 10: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/10.jpg)
How to tell if we are doing any better than existing models? Validation dataset
Concordance Index
Method Original Dutch Trial (n=459)
AJCC Stage 0.77 0.75
Nomogram 0.80 0.77
(p<0.001) (p<0.001)
![Page 11: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/11.jpg)
Heterogeneity within stages0
10
20
30
0.0 0.2 0.4 0.6 0.8 1.0
05
10
15
200
24
68
02
46
80
51
01
52
00
51
02
0
Nomogram Predicted Probability of 5-Year Disease-Specific Survival
Pe
rce
nt o
f P
atie
nts
with
in A
JC
C S
tag
e
AJCC
IV(32)
II(117)
IA(102)
IB(115)
IIIA(69)
IIIB(24)
![Page 12: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/12.jpg)
Carroll et al., J. Urol, 2004
![Page 13: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/13.jpg)
Nomograms for clinical trial design• Example: CALGB 90203, preoperative therapy for patients at high
risk of failure following surgery for prostate cancer
Biopsy Gleason Grade 2+ 2 3+3 3+ 4
2+3 4+ ?
Total Points 0 20 40 60 80 100 120 140 160 180 200
60 Month Rec. Free Prob. .96 .93 .9 .85 .8 .7 .6 .5 .4 .3 .2 .1 .05
3+ 2
Clinical Stage T1c T1ab
T2a T2c T3a
T2b
Points 0 10 20 30 40 50 60 70 80 90 100
PSA 0.1 1 2 3 6 8 9 10 12 16 30 45 70 1107 204
< 60%
![Page 14: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/14.jpg)
Continuous Models vs. Staging/Grouping Systems
Model Comparator CI (M vs C)
Preop L/I/H Risk Groups 0.67 vs. 0.64
Preop + IL6/TGFβ1
L/H Risk Groups 0.84 vs. 0.73
Pre XRT L/I/H Risk Groups 0.76 vs. 0.69
Melanoma SLN+ AJCC Stage 0.69 vs. 0.66
Pancreatic Ca AJCC Stage 0.64 vs. 0.56
Gastric Ca AJCC Stage 0.77 vs. 0.75
Breast Ca NPI Groups 0.69 vs. 0.64
Sarcoma CART Groups 0.77 vs. 0.74
![Page 15: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/15.jpg)
Software to facilitate real-time predictions
• Software is free from http://www.mskcc.org/predictiontools• Prostate, renal cell, gastric, sarcoma, breast, lung available now. • Pancreatic, melanoma available soon.
Software for the Palm Pilot, PocketPC, and Windows Desktop ComputersModels
![Page 16: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/16.jpg)
Levels of discrimination for some prediction tools
0.5 0.6 0.7 0.8 0 .9 1.0
Preoperative
Zero ability to predict
Discriminate Perfectly
Brachytherapy
Radiotherapy
Positive subsequent biopsy
Postoperative
Survival with progressive metastatic disease
Indolent Ca
LN+OC Preop with
IL-6 & TGFβ1
Sarcoma
Renal Cell
Gastric
Pancreatic
LungMelanoma
![Page 17: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/17.jpg)
When The Patient Wants A Prediction, What Options Does The Clinician Have?
• Quote an overall average to all patients
• Deny ability to predict at the individual patient level
• Assign the patient to a risk group, i.e. high, intermediate, or low
• Apply a model
• Predict based on knowledge and experience
![Page 18: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/18.jpg)
Nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients
with a positive sentinel node biopsy
Vanzee K, et al., Ann Surg Oncol., 2003.
Lobular
![Page 19: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/19.jpg)
Breast Cancer Prediction: 17 Clinicians vs. Model on 33 Patients
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
ModelCI 0.72
ClinicianCI 0.54
Sensitivity:Proportion of women withpositive nodespredicted tohave positivenodes
Specificity:Proportion ofwomen withnegative nodespredicted to havenegative nodes
![Page 20: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/20.jpg)
ROC CurvesIndividual Clinicians and Model
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
1-Specificity
Se
nsitiv
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
temp.2temp.3temp.4temp.5temp.6temp.7temp.8temp.9temp.10temp.11temp.12temp.13temp.14temp.15temp.16temp.17temp.18
Areas0.750.72 Model0.680.650.650.630.590.580.550.550.530.520.500.490.470.430.420.40
![Page 21: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/21.jpg)
Conclusions
• Concordance index is a useful metric by which to compare rival prediction models.
• The decision whether to use any model vs. assume homogeneous risk is context dependent.
![Page 22: Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center](https://reader035.vdocument.in/reader035/viewer/2022062518/56649ea85503460f94bab601/html5/thumbnails/22.jpg)
Collaborators
• Methods» Biostatistics
– Mithat Gonen– Glenn Heller– Peter Bach– Colin Begg– Frank Harrell
» Informatics– Paul Fearn– David Ladanyi– John Davey– Pat Turi– Jacob Rockowitz– Drumbeat Digital
• Applications» Peter Scardino» Murray Brennan» Marty Karpeh» Kim VanZee» Dan Coit