what can we learn from genomics being the leading
TRANSCRIPT
What can we learn from genomics being the leading
application of clinical AI?Isaac Kohane, MD, PhD
January 2019
Overview of AI’s role in healthcare delivery
• Why are so many so excited?• Where does the hype fall short?• What can we learn from the success in AI & CLINICAL Genomics
Beam, Andrew L., and Isaac S. Kohane. 2018. “Big Data and Machine Learning in Health Care.” JAMA:
The major contribution of doctors was…
Antibiotic resistance: 20th century styleSample Culture Identification Resistance
Treatment
Problems:SlowNot mechanisticNot very rich data
Antibiotic use is largely ad hoc
SpeciesResistance profile
Model for predicting TB resistance from genomic data
Drug Selected MutationsIsonaizid 19
Rifampicin 14Pyrazinamide 124
Ethambutol 18Streptomycin 39Ethionamide 20
Kanamycin 3Capreomycin 5
Amikacin 2Ciprofloxacin 7Levofloxacin 8
Ofloxacin 6p-aminosalicylic acid 4
Total 250
Farhat et al ARJCCM 2016
If AAMC puts out articles like this…
But there is rising skepticismWhere is it coming from?
Survival 3 Years After a WBC Test(White, Male, 50-69 Years; Using Last WBC Between 7/28/05 and 7/27/06)
Patient Pathophysiology Healthcare System Dynamics Data Quality
Patient DemographicsDiagnosesLaboratory Test ResultsVital SignsGenetic Markers
Number of Observations Time of Day of ObservationsTime Between ObservationsCost of a Test or TreatmentClinical Setting / Clinician Type
Data Entry ErrorsDictation MistakesData Compression LossUnstructured DataMissing Data
Patient Pathophysiology
Healthcare System Dynamics
Normal Abnormal
Normal Best Outcomes
Moderate Outcomes
Abnormal Moderate Outcomes
Worst Outcomes
Healthcare System Dynamics
EHR
Clinician
Electronic Health Record Data
ClinicalEncounter
Clinical data reflect both patients’ health AND their interactions with the healthcare system.
Patient
Predicting Survival from Ordering a Lab TestHigher Mortality RateHigher Survival Rate
Predicting Survival Using Lab Value & HSDHSD More PredictiveValue More Predictive
• Who completes the loop?• Who is trusted?• Who integrates with rest of care?
• Who oversees• Process automation?
• Who does expert catch?
Contrast…
16
Worsening dystonia(not walking, not speaking)
L-DopaFolinic Acid
5-hydroytryptophan
GTP cyclohydrolase I deficiency
Walking, talking!
Trying to make rare more common.
Why is genetics privileged?
• Large body of rapidly evolving knowledge• Millions of variants per person (and dozens with textbook
associations with disease)
Authoritative interpretation?
22
!
0.0 0.2 0.4 0.6 0.8 1.0
05
1015
2025
30Survey Responses
Response
Cou
nt
n = 61
● ●●● ●
●●●
●
●
●
●●●●
● ●
●●
●
●
●
●
●●
●●●
●
●
●●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●●
●
Correct answer − 2%n = 14
Most common answer − 95%n = 27
●
●
●
StudentsHouse StaffAttendings
n = 10n = 26n = 25
Figure'1:'Distribution!of!responses!to!survey!question!provided!in!the!article!text.!
14!of!61!respondents!provided!the!correct!answer!of!2%.!The!most!common!answer!
was!95%,!provided!by!27!of!61!respondents.!The!median!answer!was!66%,!which!is!
33!times!larger!than!the!true!answer.'
'
'
Copyright 2014 American Medical Association. All rights reserved.
Letters
RESEARCH LETTER
Medicine’s Uncomfortable RelationshipWithMath:Calculating Positive Predictive ValueIn 1978, Casscells et al1 published a small but important studyshowing that the majority of physicians, house officers, andstudents overestimated the positive predictive value (PPV) ofa laboratory test resultusingprevalenceand falsepositive rate.
Today, interpretationofdiag-nostic tests is evenmorecriti-calwith the increasinguse of
medical technology in health care. Accordingly, we repli-cated the study by Casscells et al1 by asking a conveniencesample of physicians, house officers, and students the samequestion: “If a test to detect a disease whose prevalence is1/1000 has a false positive rate of 5%, what is the chance thata person found to have a positive result actually has the dis-ease, assuming you know nothing about the person's symp-toms or signs?”
Methods |During July 2013,wesurveyedaconvenience sampleof 24 attending physicians, 26 house officers, 10medical stu-dents, and 1 retiredphysician at aBoston‐areahospital, acrossa wide range of clinical specialties (Table). Assuming a per-fectly sensitive test, we calculated that the correct answer is1.96% and considered “2%,” “1.96%,” or “<2%” correct. 95%Confidence intervals were computed using the exact bino-mial and 2‐sample proportion functions in R. The require-ment for study approval was waived by the institutionalreview board of Department of Veterans Affairs BostonHealthcare System.
Results | Approximately three-quarters of respondents an-swered the question incorrectly (95% CI, 65% to 87%). In ourstudy, 14 of 61 respondents (23%) gave a correct response, notsignificantlydifferent fromthe 11of60correct responses (18%)in the Casscells study (difference, 5%; 95% CI, −11% to 21%).In both studies themost commonanswerwas “95%,” givenby27 of 61 respondents (44%) in our study and 27 of 60 (45%) inthe study by Casscells et al1 (Figure). We obtained a range ofanswers from“0.005%”to“96%,”withamedianof66%,whichis 33 times larger than the true answer. In brief explanationsof their answers, respondents oftenknew to computePPVbutaccounted for prevalence incorrectly. For example, one at-tendingcardiologistwrote that“PPVdoesnotdependonpreva-lence,” and a resident wrote “better PPV when prevalence islow.”
Discussion |Withwider availability ofmedical technology anddiagnostic testing, sound clinical management will increas-ingly depend on statistical skills. Wemeasured a key facet ofstatistical reasoning inpracticingphysicians and trainees: the
evaluation of PPV. Understanding PPV is particularly impor-tantwhenscreening forunlikely conditions,whereevennomi-nally sensitive and specific tests can be diagnostically unin-formative. Our results show that themajority of respondentsin this single-hospital study could not assess PPV in the de-scribedscenario.Moreover, themost commonerrorwasa largeoverestimation of PPV, an error that could have considerableimpact on the course of diagnosis and treatment.
Editor's Note Table. Survey Respondentsa
Level of Training
No. of Respondents
Casscells et al1 Present StudyMedical student 20 10
Intern
20b
12
Resident 8
Fellow 6
Attending physician 20 24
Retired 0 1
Total 60 61
a This table gives the breakdown of the physicians and trainees surveyed in ourstudy and the study of Casscells et al.1 The study by Casscells et al wasperformed at Harvard Medical School in 1978. Our study included Harvard andBoston University medical students along with residents and attendingphysicians affiliated with these 2medical schools. Of the 30 fellows andattending physicians, the most represented specialties were internal medicine(n = 10), cardiology (n = 4), spinal cord injury (n = 2), pulmonology (n = 2),and psychiatry (n = 2), with 1 attending physician or fellow from each of 8other specialties.
b Casscells et al1 split their sample into students, house officers, and attendingphysicians. They did not break down the house officers category further.
Figure. Distribution of Responses to Survey Question Provided in theArticle Text
30
25
20
15
10
5
0
0 10080
Surv
ey R
espo
nden
ts, N
o.
Response, %
Correct answer: 2%
Most common answer: 95%
20 40 60
Students (n = 10)House staff (n = 26)Attending physicians (n = 25)
Of 61 respondents, 14 provided the correct answer of 2%. Themost commonanswer was 95%, provided by 27 of 61 respondents. Themedian answer was66%, which is 33 times larger than the true answer.
jamainternalmedicine.com JAMA Internal Medicine Published online April 21, 2014 E1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archinte.jamanetwork.com/ by a Harvard University User on 04/23/2014
Copyright 2014 American Medical Association. All rights reserved.
Letters
RESEARCH LETTER
Medicine’s Uncomfortable RelationshipWithMath:Calculating Positive Predictive ValueIn 1978, Casscells et al1 published a small but important studyshowing that the majority of physicians, house officers, andstudents overestimated the positive predictive value (PPV) ofa laboratory test resultusingprevalenceand falsepositive rate.
Today, interpretationofdiag-nostic tests is evenmorecriti-calwith the increasinguse of
medical technology in health care. Accordingly, we repli-cated the study by Casscells et al1 by asking a conveniencesample of physicians, house officers, and students the samequestion: “If a test to detect a disease whose prevalence is1/1000 has a false positive rate of 5%, what is the chance thata person found to have a positive result actually has the dis-ease, assuming you know nothing about the person's symp-toms or signs?”
Methods |During July 2013,wesurveyedaconvenience sampleof 24 attending physicians, 26 house officers, 10medical stu-dents, and 1 retiredphysician at aBoston‐areahospital, acrossa wide range of clinical specialties (Table). Assuming a per-fectly sensitive test, we calculated that the correct answer is1.96% and considered “2%,” “1.96%,” or “<2%” correct. 95%Confidence intervals were computed using the exact bino-mial and 2‐sample proportion functions in R. The require-ment for study approval was waived by the institutionalreview board of Department of Veterans Affairs BostonHealthcare System.
Results | Approximately three-quarters of respondents an-swered the question incorrectly (95% CI, 65% to 87%). In ourstudy, 14 of 61 respondents (23%) gave a correct response, notsignificantlydifferent fromthe 11of60correct responses (18%)in the Casscells study (difference, 5%; 95% CI, −11% to 21%).In both studies themost commonanswerwas “95%,” givenby27 of 61 respondents (44%) in our study and 27 of 60 (45%) inthe study by Casscells et al1 (Figure). We obtained a range ofanswers from“0.005%”to“96%,”withamedianof66%,whichis 33 times larger than the true answer. In brief explanationsof their answers, respondents oftenknew to computePPVbutaccounted for prevalence incorrectly. For example, one at-tendingcardiologistwrote that“PPVdoesnotdependonpreva-lence,” and a resident wrote “better PPV when prevalence islow.”
Discussion |Withwider availability ofmedical technology anddiagnostic testing, sound clinical management will increas-ingly depend on statistical skills. Wemeasured a key facet ofstatistical reasoning inpracticingphysicians and trainees: the
evaluation of PPV. Understanding PPV is particularly impor-tantwhenscreening forunlikely conditions,whereevennomi-nally sensitive and specific tests can be diagnostically unin-formative. Our results show that themajority of respondentsin this single-hospital study could not assess PPV in the de-scribedscenario.Moreover, themost commonerrorwasa largeoverestimation of PPV, an error that could have considerableimpact on the course of diagnosis and treatment.
Editor's Note Table. Survey Respondentsa
Level of Training
No. of Respondents
Casscells et al1 Present StudyMedical student 20 10
Intern
20b
12
Resident 8
Fellow 6
Attending physician 20 24
Retired 0 1
Total 60 61
a This table gives the breakdown of the physicians and trainees surveyed in ourstudy and the study of Casscells et al.1 The study by Casscells et al wasperformed at Harvard Medical School in 1978. Our study included Harvard andBoston University medical students along with residents and attendingphysicians affiliated with these 2medical schools. Of the 30 fellows andattending physicians, the most represented specialties were internal medicine(n = 10), cardiology (n = 4), spinal cord injury (n = 2), pulmonology (n = 2),and psychiatry (n = 2), with 1 attending physician or fellow from each of 8other specialties.
b Casscells et al1 split their sample into students, house officers, and attendingphysicians. They did not break down the house officers category further.
Figure. Distribution of Responses to Survey Question Provided in theArticle Text
30
25
20
15
10
5
0
0 10080
Surv
ey R
espo
nden
ts, N
o.
Response, %
Correct answer: 2%
Most common answer: 95%
20 40 60
Students (n = 10)House staff (n = 26)Attending physicians (n = 25)
Of 61 respondents, 14 provided the correct answer of 2%. Themost commonanswer was 95%, provided by 27 of 61 respondents. Themedian answer was66%, which is 33 times larger than the true answer.
jamainternalmedicine.com JAMA Internal Medicine Published online April 21, 2014 E1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archinte.jamanetwork.com/ by a Harvard University User on 04/23/2014
- Heart failure- Arrhythmias- Obstructed blood flow- Infective endocarditis- Sudden cardiac death
Prevalence 1:500Autosomal Dominant
Hypertrophic Cardiomyopathy (HCM)
expected100 individuals
HCM Prevalence = 1:500HCM Inheritance = Autosomal Dominant
NHLBI ESP6503 individuals
observed
NHLBI ESP6503 individuals
classified as pathogenic 2005-2007
0
0.02
0.04
0.06
0.08
0.1
0.12
TNNT2 (K247R) OBSCN(R4344Q) TNNI3 (P82S) MYBPC3
(G278E) JPH2 (G505S) Remaining 89mutations
0.11102568 0.052749361 0.012853005 0.010360138 0.01410141 0.077135379
min
or a
llele
gen
otyp
e pr
eval
ence
Age$ Ethnicity$ Report$Year$
Originally$Reported$Status$
Current$Status$ Indication$for$Test$
46# Unavailable# 2005# P# B# Clinical#Diagnosis#of#HCM#
75# Unavailable# 2005# P# B#Family#History#and#Clinical#
Symptoms#of#HCM#
32# Black#or#African#American# 2005# P# B# Clinical#Diagnosis#of#HCM#
34# Black#or#African#American# 2005# U# B#Clinical#Diagnosis#and#Family#History#of#HCM#
12# Black#or#African#American# 2006# U# B# Family#History#of#HCM#
40# Black#or#African#American# 2007# U# B# Clinical#Diagnosis#of#HCM#
45# Black#or#African#American# 2007# U# B# Clinical#Features#of#HCM#
16# Asian# 2008# U# B#Clinical#Diagnosis#and#Family#History#of#HCM#
59# Black#or#African#American# 2006# P# B# Clinical#Features#of#HCM#
15# Black#or#African#American# 2007# P# B# Clinical#Diagnosis#of#HCM#
16# Black#or#African#American# 2007# P# B# Clinical#Diagnosis#of#HCM#
22# Black#or#African#American# 2007# P# B#Clinical#Diagnosis#and#Family#History#of#HCM#
48# Black#or#African#American# 2008# U# B# Clinical#Diagnosis#of#HCM#
P"="Pathogenic"and"Presumed"Pathogenic"U"="Pathogenicity"Debated"and"Unknown"Significance""
Pro82Ser
Gly278Glu
• Make sure the
accounting is sound.
How are we going to get there?
•Liberate our data•Upgrade our healthcare systems and care givers
SMART Vision
• Write codes onc for use in many clinical platforms
• Eliminate barriers for clinical IT developers to
• Create clinically specific medical UX• Integrate medical knowledge faster• Respond promptly to market needs and wants
• Increase ROI in EMR/clinical systems
• Stimulate supply to address demand
28
When the academic industrial collaboration around data science redefines the possible.
http://www.decaturcountyhospital.org
Dignity Health (Arizona, California, and Nevada) https://www.dignityhealth.org
Duke Health (North Carolina)http://dukehealth.org
Eisenhower Health (California)https://www.eisenhowerhealth.org
Family Foot and Ankle Clinic, P.A. (Minnesota)https://www.familyfootmn.com
Family Footcare Specialist (New Jersey)https://familyfootcarespecialist.com
Fisher-Titus Health (Ohio)https://www.fishertitus.org
Fitzgibbon Hospital (Missouri)https://www.fitzgibbon.org
Foot and Ankle Specialist (Arizona)
Fort Healthcare (Wisconsin)https://www.forthealthcare.com
Franciscan Missionaries of Our Lady Health System (Louisiana)https://fmolhs.org
Geisinger (Pennsylvania)http://geisinger.org
Genesis Healthcare System (Ohio) https://www.genesishcs.org
Grace Cottage Hospital (Vermont)https://gracecottage.org
Greater Hudson Valley Health System (New York)https://www.ormc.org/about-us/ghvhs
Greenville Health System (South Carolina)https://www.ghs.org
Group Health Cooperative of South Central Wisconsin (Wisconsin)https://ghcscw.com
Adult Internal Medicine (North Carolina)http://www.adultinternalmedicine.net
Adventist Health System (Colorado, Florida, Georgia, Illinois, Texas, and others)https://www.adventisthealthsystem.com
AtlantiCare (New Jersey)http://atlanticare.org
Austin Regional Clinic (Texas)https://www.austinregionalclinic.com
Dan Bangart, DPM (Arizona)https://peoriafootandanklespecialists.com
Baptist Health (Kentucky, Indiana, and Illinois)https://www.baptisthealth.com/pages/home.aspx
BayCare (Florida)https://baycare.org
Better Me Healthcare (Florida)http://www.bettermehealthcare.com
Black River Memorial Hospital (Wisconsin)https://www.brmh.net
Bon Secours Health System (Kentucky, Maryland, South Carolina, Virginia, and Florida)https://bonsecours.com
Boone County Health Center (Nebraska)https://boonecohealth.org
Dr. Albert Boyd, MD (Texas)https://www.alboydmd.com
Brain Vitals (California)
Buffalo Medical Group (New York)https://www.buffalomedicalgroup.com
Carroll County Memorial Hospital (Missouri)http://www.carrollcountyhospital.org
Carson Valley Medical Center (Nevada)https://cvmchospital.org
Cedars-Sinai (California)https://www.cedars-sinai.org
Center for Manual Medicine (Kansas)http://www.ctrmm.com
Central Valley Medical Center (Utah)https://www.centralvalleymedicalcenter.com
Centura Health (Colorado and Kansas)https://www.centura.org
Cerner Healthe Clinic (Kansas, Missouri)https://healtheatcerner.com
Chase County Community Hospital (Nebraska)https://www.chasecountyhospital.com
Christiana Care Health System (Delaware, Pennsylvania, Maryland, and New Jersey)https://christianacare.org
Circle Health (Massachusetts)http://www.circle-health.org
Cleveland Clinic (Ohio, Nevada, and Florida)https://my.clevelandclinic.org
Cogdell Memorial Hospital (Texas)https://cogdellhospital.com
Cone Health (North Carolina)https://www.conehealth.com
Confluence Health (Washington)https://www.confluencehealth.org
Coquille Valley Hospital (Oregon)http://www.cvhospital.org
CoxHealth (Missouri)https://www.coxhealth.com
David J. MacGregor MD (California)http://450derm.com
Deaconess Health System (Indiana)https://www.deaconess.com
http://www.decaturcountyhospital.org
Dignity Health (Arizona, California, and Nevada) https://www.dignityhealth.org
Duke Health (North Carolina)http://dukehealth.org
Eisenhower Health (California)https://www.eisenhowerhealth.org
Family Foot and Ankle Clinic, P.A. (Minnesota)https://www.familyfootmn.com
Family Footcare Specialist (New Jersey)https://familyfootcarespecialist.com
Fisher-Titus Health (Ohio)https://www.fishertitus.org
Fitzgibbon Hospital (Missouri)https://www.fitzgibbon.org
Foot and Ankle Specialist (Arizona)
Fort Healthcare (Wisconsin)https://www.forthealthcare.com
Franciscan Missionaries of Our Lady Health System (Louisiana)https://fmolhs.org
Geisinger (Pennsylvania)http://geisinger.org
Genesis Healthcare System (Ohio) https://www.genesishcs.org
Grace Cottage Hospital (Vermont)https://gracecottage.org
Greater Hudson Valley Health System (New York)https://www.ormc.org/about-us/ghvhs
Greenville Health System (South Carolina)https://www.ghs.org
A-GFromhttps://support.apple.com/en-us/HT208647
371 hospitals as of10/19/2019
The young doctors-to be-KNOW that medicine is an information processing discipline
Thank you