clinical prediction models in the age of artificial...
TRANSCRIPT
![Page 1: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/1.jpg)
Clinical prediction models in the age of artificial intelligence and big data
Ewout SteyerbergProfessor of Clinical Biostatistics and Medical
Decision Making
Basel, Nov 1 2019
![Page 2: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/2.jpg)
Thanks to co-workers; no COI
• LUMC: Maarten van Smeden
• Leuven: Ben van Calster
Both provided many of the slides shown
![Page 3: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/3.jpg)
Main question
Where does Big Data / machine learning (ML) / artificial intelligence (AI) assist us in prediction research?
• Strengths and weaknesses of Big Data initiatives
• Consider links between classical statistical approaches, ML, AI for prediction
![Page 4: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/4.jpg)
Prediction models; what for?
• Understanding nature: relative risks of different predictors
• Predicting outcomes: absolute risk by combinations of predictors
![Page 5: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/5.jpg)
Traditional regression modeling
5
Can well be used for explanation and prediction
Steyerberg. Clinical prediction models (2nd ed). New York: Springer, 2019.Riley et al. Prognosis Research in healthcare. Oxford: OUP, 2019.
![Page 6: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/6.jpg)
Prediction models
• Diagnosis
– Imaging findings, e.g. abnormal CT scan in trauma
– Clinical condition, e.g. serious infection
– …
• Prognosis
– Mortality, e.g. < 30 days, over time, …
– …
![Page 7: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/7.jpg)
Prognostic / predictive models
Prognostic modeling
y ~ X Prognostic factors
y ~ Tx Treatment effect
y ~ X + Tx Covariate adjusted tx effect
Predictive modeling
y ~ X * Tx Predictive factors for differential tx effect
![Page 8: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/8.jpg)
Opportunities in medical prediction
• More data
– larger N
– more variables
• More detail
– biomarkers / omics / imaging / eHealth
• Novel methods
– ML / AI / ..
– Statistical methods
• Dynamic prediction
• Testing procedures for high dimensional data
• …
![Page 9: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/9.jpg)
Hype
![Page 10: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/10.jpg)
Examples
• Biomarkers
• Imaging
• Omics
![Page 11: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/11.jpg)
Positive example 1
• Biomarkers in diagnosing head trauma
– Mild: AUC 0.89 [0.87-0.90] vs clinical 0.84 [0.83-0.86]
![Page 12: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/12.jpg)
Positive example 2
• MRI Imaging in diagnosing prostate cancer
• MRI-PCa-RCs AUC 0.83 to 0.85 vsPCa-RCs AUC 0.69 to 0.74
![Page 13: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/13.jpg)
Positive example 3
![Page 14: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/14.jpg)
Positive example 3
• Omics in diagnosing … / predicting … ??
• Because omics clinical characteristics
outcome?
![Page 15: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/15.jpg)
Examples
• Biomarkers
• Imaging
• Omics
• ML / AI
![Page 16: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/16.jpg)
Success of ML / AI
![Page 17: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/17.jpg)
Non-exhaustive list
17
Gaming
Natural Language Processing (Siri etc)
Fraud detection
Shoplifting
Object recognition (e.g. for driverless cars)
Facial recognition
Traffic predictions (e.g. Waze app)
Electrical load forecasting
(Social) media and advertising (people you may know, movie suggestions, )
Spam filtering
Search engines (e.g. Google PageRank)
Handwriting recognition
![Page 18: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/18.jpg)
Popularity skyrocketing
18
Search on https://www.ncbi.nlm.nih.gov/pubmed/ on (performed Oct 18, 2019)
![Page 19: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/19.jpg)
IBM Watson winning Jeopardy! (2011)
![Page 20: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/20.jpg)
IBM Watson for oncology
https://bit.ly/2LxiWGj
![Page 21: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/21.jpg)
Evidence
• Cochrane: ”We searched for RCTs and found 20 among ... papers”
• Dr Watson: “We searched 4 Million webpages in 1 second”
![Page 22: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/22.jpg)
Five myths
1. Big Data will resolve the problems of small data
2. ML/AI is very different from classical modeling
3. Deep learning is relevant for all medical prediction problems
4. ML / AI is better than classical modeling for medical prediction problems
5. ML / AI leads to better generalizability
![Page 23: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/23.jpg)
Myth 1: Big Data will resolve theproblems of small data
![Page 24: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/24.jpg)
Abstract
The use of artificial intelligence, and deep-learning in particular, has been enabled by the use of big data, along with markedly enhanced computing power and cloud storage, across all sectors.
In medicine, this is beginning to have an impact ...
![Page 25: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/25.jpg)
Do you have a clear research question?Do you have data that help you answer the question?
What is the quality of the data?
![Page 26: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/26.jpg)
Do you have a clear research question?Do you have data that help you answer the question?
What is the quality of the data?
![Page 27: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/27.jpg)
Do you have a clear research question?Do you have data that help you answer the question?
What is the quality of the data?
![Page 28: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/28.jpg)
Big Data, Big Errors
• Harrell tweet
![Page 29: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/29.jpg)
Myth 2: ML/AI is very differentfrom classical modeling
![Page 30: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/30.jpg)
“Everything is ML”
https://bit.ly/2lEVn33
![Page 31: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/31.jpg)
Two cultures
Breiman, Stat Sci, 2001, DOI: 10.1214/ss/1009213726
![Page 32: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/32.jpg)
Traditional Statistics vs Machine Learning
32
Breiman. Stat Sci 2001;16:199-231.
![Page 33: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/33.jpg)
Traditional Statistics vs Machine Learning
33Galit Shmueli. Keynote talk at 2019 ISBIS conference, Kuala Lumpur; taken from slideshare.netBzdok. Nature Methods 2018;15:233-4.
??
![Page 34: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/34.jpg)
Example of exaggerating contrasts
![Page 35: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/35.jpg)
![Page 36: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/36.jpg)
Predicting mortality – the results
Elastic net, 586 (‘600’) variables: c=0.801
Traditional Cox, 27 (‘30’) expert-selected variables: c=0.793
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344
![Page 37: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/37.jpg)
Predicting mortality – the media
PlosOne, 2018, DOI: 10.1371/journal.pone.0202344; https://bit.ly/2Q6H41R; https://bit.ly/2m3RLrn
![Page 38: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/38.jpg)
ML refers to a culture, not to methods
• Substantial overlap methods used by both cultures
• Substantial overlap analysis goals
• Attempts to separate the two frequently result in disagreement
Pragmatic approach:
“ML” refers to models roughly outside of the traditional regression types of analysis: trees, SVMs, neural networks, boosting etc.
![Page 39: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/39.jpg)
Machine learning: simple overview
39
Intellspot.com
![Page 40: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/40.jpg)
Myth 3: Deep learning is relevantfor all medical prediction
![Page 41: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/41.jpg)
![Page 42: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/42.jpg)
Example: retinal disease
Gulshan et al, JAMA, 2016, 10.1001/jama.2016.17216; Picture retinopathy: https://bit.ly/2kB3X2w AS
Diabetic retinopathy
Deep learning (= Neural network)
• 128,000 images
• Transfer learning (preinitialization)
• Sensitivity and specificity > .90• Estimated from training data
![Page 43: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/43.jpg)
Example: lymph node metastases
Bejnordi et al, JAMA, 2018, doi: 10.1001/jama.2017.14585. See letter to the editor for a critical discussion: https://bit.ly/2kcYS0e
Deep learning competition
But:
• 390 teams signed up, 23 submitted
• “Only” 270 images for training
• Test AUC range: 0.56 to 0.99
![Page 44: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/44.jpg)
3. Deep learning is relevant for all medical prediction problemsNO: Deep learning excels in visual tasks
![Page 45: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/45.jpg)
Myth 4: ML / AI is better than classicalmodeling for medical prediction
![Page 46: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/46.jpg)
Reviewer #2, van Smeden submission 2019
![Page 47: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/47.jpg)
![Page 48: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/48.jpg)
Poor methods and unclear reporting
48
What was done about missing data? 45% fully unclear, 100% poor or unclear
How were continuous predictors modeled? 20% unclear, 25% categorized
How were hyperparameters tuned? 66% unclear, 19% tuned with information
How was performance validated? 68% unclear or biased approach
Was accuracy of risk estimates checked? 79% not at all
Further observations:- Prognosis: time horizon often ignored- Patients matched on variables used a predictors- 99% of patients excluded from modeling to obtain a balanced dataset- First and last percentile of continuous predictors replaced with mean
![Page 49: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/49.jpg)
Differences in discrimination
Christodoulou et al. Journal of Clinical Epidemiology, 2019, doi: 10.1016/j.jclinepi.2019.02.004
![Page 50: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/50.jpg)
![Page 51: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/51.jpg)
Where is ML useful?
![Page 52: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/52.jpg)
![Page 53: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/53.jpg)
Rajkomar et al. NEJM 2019;380:1347-58.
![Page 54: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/54.jpg)
Myth 5: ML / AI leads to better generalizability
“ … developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.”:
“Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration … ”
![Page 55: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/55.jpg)
Efron talk Leiden
![Page 56: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/56.jpg)
![Page 57: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/57.jpg)
Empirical findings in TBI
– 16 cohorts: 5 observational, 11 RCTs
– Develop in 15, validate in 1
– 7 methods: LR; SVM; RF; nnet; gbm; LASSO; ridge
![Page 58: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/58.jpg)
5 observational 11 RCTs
Variability between cohorts >> variability between methods
![Page 59: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/59.jpg)
Prediction challenges
• There is no such thing as a validated prediction algorithm
• Algorithms are high maintenance
– Developed models need validation and updating to remain useful over time and place
• Regulation and quality control of algorithms
– What about proprietary algorithms?
![Page 60: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/60.jpg)
Five myths1. Big Data will resolve the problems of small data
NO: Big Data, Big Errors
2. ML/AI is very different from classical modelingNO: a continuum, cultural differences
3. Deep learning is relevant for all medical predictionNO: Deep learning excels in visual tasks
4. ML / AI is better than classical modeling for predictionNO: some methods do harm (e.g. tree modeling)
5. ML / AI leads to better generalizabilityNO: any prediction model may suffer from poor generalizability
![Page 61: Clinical prediction models in the age of artificial ...bbs.ceb-institute.org/wp-content/uploads/2019/08/1_Steyerberg.pdf · –biomarkers / omics / imaging / eHealth •Novel methods](https://reader033.vdocument.in/reader033/viewer/2022052013/602a2d9105d3c40ccb6fa4f1/html5/thumbnails/61.jpg)