50 shades of grey - australian and new zealand college of ... · 50 shades of grey how (in)accurate...
TRANSCRIPT
![Page 1: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/1.jpg)
50 shades of greyhow (in)accurate are risk prediction scores
Dr Doug CampbellMeasuring, managing and minimising riskPerioperative SIG, Grand Hyatt, Melbourne
October 27th 2018
![Page 2: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/2.jpg)
Declarations and Acknowledgements
• No relevant conflicts of interest• Funded by Precision Driven Health
![Page 3: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/3.jpg)
![Page 4: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/4.jpg)
![Page 5: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/5.jpg)
Those who make many species are the 'splitters,' and those who make few are the 'lumpers.'
![Page 6: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/6.jpg)
Myth of individualised prediction
…with apologies to XKCD
![Page 7: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/7.jpg)
Validation 101- easy as ABCD
![Page 8: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/8.jpg)
Validation 101- easy as ABCD
A=0B=0.96C=0.922
![Page 9: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/9.jpg)
nzRISK – CalibrationIntercept = 0Slope = 0.97McFadden’s R2 = 0.28AUROC = 0.92
Intercept = 0Slope = 0.96McFadden’s R2 = 0.31AUROC = 0.91
Intercept = 0Slope = 0.98McFadden’s R2 = 0.37AUROC = 0.90
1 month
1 year 2 year
![Page 10: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/10.jpg)
nzRISK – DiscriminationCentile Total Predicted Observed Predicted Observed
N (%) N % % N N
1 9481 0 0 0.9 1
2 8528 0 0 1.5 0
3 9303 0 0 2.9 1
4 8713 0.1 0.1 5.4 5
5 9026 0.1 0.1 9.5 5
6 9192 0.2 0.1 14.8 8
7 (0.2-0.3) 8784 0.2 0.3 21.6 26
8 (0.3—0.5) 9008 0.4 0.2 35.9 21
9 (0.5 – 1.5) 9004 0.9 0.9 77.5 78
10 (>1.5) 8996 5.6 5.4 508.1 489
![Page 11: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/11.jpg)
nzRISK – DiscriminationCentile Total Predicted Observed Predicted Observed
N (%) N % % N N
1 9481 0 0 0.9 1
2 8528 0 0 1.5 0
3 9303 0 0 2.9 1
4 8713 0.1 0.1 5.4 5
5 9026 0.1 0.1 9.5 5
6 9192 0.2 0.1 14.8 8
7 (0.2-0.3) 8784 0.2 0.3 21.6 26
8 (0.3—0.5) 9008 0.4 0.2 35.9 21
9 (0.5 – 1.5) 9004 0.9 0.9 77.5 78
10 (>1.5) 8996 5.6 5.4 508.1 489
Low risk, NZRISK < 0.5%, 11% of deaths
High risk, NZRISK > 1.5%, 77% of deaths
Intermediate risk, NZRISK 0.5-1.5%, 12% of deaths
![Page 12: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/12.jpg)
Decision curve analysis
Glance et al. Impact of the choice of model for identifying low-risk patients using the 2014 ACC/AHA perioperative guidelines. Anesthesiology 2018; 129(5): 889-900
![Page 13: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/13.jpg)
Generalisability
SORT internal validation
AUROC = 0.91
AUROC = 0.90
SORT external validation
![Page 14: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/14.jpg)
Generalisability
SORT internal validation
AUROC = 0.91
AUROC = 0.90
SORT external validation
Protopapa KL et al. Development and validation of the Surgical Outcome Risk Tool (SORT). Br J Surgery 2014; 101: 1774-83Campbell D et al. Development and validation of a multivariable prediction model of perioperative mortality in noncardiac surgery (NZRISK). Br J Surgery 2018; in press
![Page 15: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/15.jpg)
What risk tool for CADENZA?
Eugene N et al. Development of the NELA risk model. BJA 2018; 121(4): 739-48
![Page 16: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/16.jpg)
ACS-NSQIP - morbidity
![Page 17: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/17.jpg)
DAH90 – the median is not the message
Campbell D, Boyle L, Djamali K, Cumin D, Weller J, Short T, Merry A. Unpublished data.
New Zealand 2011-6, n = 141331-month mortality = 4.2%3-month mortality = 5.7%
83 days48 days
![Page 18: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/18.jpg)
Checklist for risk tools
• Does it model the correct outcome?– 1-month mortality, morbidity, long-term mortality
• Is it accurate?– calibration (SHARED DECISION-MAKING)– discrimination (TRIAGE)
• Is it generalizable to my patient?– validated in Australia or New Zealand?
![Page 19: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/19.jpg)
www.nzrisk.com
![Page 20: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/20.jpg)
www.nzrisk.com
![Page 21: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/21.jpg)
www.nzrisk.com
![Page 22: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/22.jpg)
![Page 23: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/23.jpg)
Comparison - general
Calculator Covariates Country N AUROC Calibration
ACS-NSQIP 22 US 1.4 million 0.944 Brier 0.011POSPOM 17 France 5.5 million 0.929 Brier 0.004nzRISK 8 NZ 360140 0.922 Brier 0.007SORT 6 UK 16,788 0.91 H-L 12.16P-POSSUM 18 UK 10,648 total 0.83 (0.68-0.92)
H-L = Hosmer-Lemeshow statistic, McF = McFaddens pseudo rho-squared statistic, Brier = Brier score, O:E observed to expected ratio
![Page 24: 50 shades of grey - Australian and New Zealand College of ... · 50 shades of grey how (in)accurate are risk prediction scores Dr Doug Campbell Measuring, managing and minimising](https://reader033.vdocument.in/reader033/viewer/2022041620/5e3ebc94b641503fdc6855d9/html5/thumbnails/24.jpg)
High risk calibration