Upload File

direct uncertainty prediction for medical second opinions12-11...poster #246: direct uncertainty...

  • Home
  • Documents
  • Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given
19
Poster #246: Direct Uncertainty Prediction for Medical Second Opinions Direct Uncertainty Prediction for Medical Second Opinions Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Sendhil Mullainathan, Jon Kleinberg Poster #246

Upload: others

Post on 02-Jan-2021

3 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Direct Uncertainty Prediction for Medical Second Opinions

Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Sendhil Mullainathan, Jon Kleinberg

Poster #246

Page 2: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Human Expert Disagreements

Page 3: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Human Expert Disagreements

Page 4: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Doctor Disagreements

Page 5: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Doctor DisagreementsDiagnostic Concordance Amongst Pathologists Interpreting Breast Biopsy Specimens, UW School of Medicine, JAMA, 2015

Page 6: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Doctor DisagreementsDiagnostic Concordance Amongst Pathologists Interpreting Breast Biopsy Specimens, UW School of Medicine, JAMA, 2015

● Agreement between individual pathologist grade and a panel consensus score on ~240 breast biopsies, 6900 individual case diagnoses

● 25% disagreement between pathologists and consensus

Page 7: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Doctor Disagreements

Grade 3: Moderate Diabetic Retinopathy

Grade 2: Mild Diabetic Retinopathy

Ophthalmology: Diagnosis from Fundus Photographs

Page 8: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

The Source of Disagreements

Grade 3: Moderate Diabetic Retinopathy

Grade 2: Mild Diabetic Retinopathy

Random Mistakes?

Ophthalmology: Diagnosis from Fundus Photographs

Page 9: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

The Source of Disagreements

Diagnosis TypeDiagnosis Type

Frac

tion

of v

otes

Patient 1 Patient 2

Page 10: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

ML for Doctor Disagreement PredictionGiven input (image) x, predict the amount of disagreement. Flag patients for medical second opinions.

Page 11: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

ML for Doctor Disagreement PredictionGiven input (image) x, predict the amount of disagreement. Flag patients for medical second opinions.

Training data: xi, with multiple labels y(i)1,...,y

(i)k (different

doctors) I.e. (xi, pi), pi grade distribution, target U(pi) (e.g. U() = entropy)

Page 12: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

ML for Doctor Disagreement PredictionGiven input (image) x, predict the amount of disagreement. Flag patients for medical second opinions.

Training data: xi, with multiple labels y(i)1,...,y

(i)k (different

doctors) I.e. (xi, pi), pi grade distribution, target U(pi) (e.g. U() = entropy)

1) Uncertainty Via Classification (UVC): (i) train classifier on empirical distribution of labels (xi, pi) (ii) postprocess with U()

2) Direct Uncertainty Prediction (DUP): directly predict scalar uncertainty score (xi, U(pi))

Page 13: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Direct Uncertainty Prediction

Page 14: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Direct Uncertainty PredictionHidden information:

61 (age) F (gender) medical history

Page 15: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Direct Uncertainty Prediction

Theorem: DUP gives an unbiased estimate of true uncertainty

Page 16: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Empirical Results: Synthetic Examples

Mixture of Gaussians

SVHN and CIFAR-10: Image Blurring Application

Page 17: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Large Scale Medical Application

Diabetic Retinopathy (DR)5 class scale:

1 None2 Mild

3 Moderate4 Severe5 Proliferative

Referable

Page 18: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Large Scale Medical Application

Page 19: Direct Uncertainty Prediction for Medical Second Opinions12-11...Poster #246: Direct Uncertainty Prediction for Medical Second Opinions ML for Doctor Disagreement Prediction Given

Poster #246: Direct Uncertainty Prediction for Medical Second Opinions

Large Scale Medical ApplicationSmall Gold Standard Evaluation SetIndividual Grades by Specialists

3 2 2 3Single, Consensus, Adjudicated Grade

Poster #246


Uncertainty Quantification in Wind Power Prediction · Uncertainty Quantification in Wind Power Prediction Jeroen Witteveen, Center for Mathematics and Computer Science (CWI), Amsterdam,

Prediction under Uncertainty in Sparse Spectrum Gaussian …proceedings.mlr.press/v70/pan17a/pan17a.pdf · 2020. 8. 18. · Prediction under Uncertainty in Sparse Spectrum Gaussian

On the Prediction of Uncertainty in a Sediment …...Research Article On the Prediction of Uncertainty in a Sediment Provenance Model Ocimum Scientific Publishers. 01. Submit Manuscript

UNCERTAINTY REVISITED: LEGAL PREDICTION AND LEGAL POSTDICTION

Online Prediction under Model Uncertainty Via Dynamic Model Averaging: Application … · 2011. 5. 14. · Online Prediction under Model Uncertainty Via Dynamic Model Averaging: Application

Uncertainty estimation and prediction for interdisciplinary ...mseas.mit.edu/publications/PDF/lermusiaux_jcp_2006.pdfUncertainty estimation and prediction for interdisciplinary ocean

Online Prediction under Model Uncertainty Via Dynamic

Uncertainty estimation in chemometrics and downloads... · Uncertainty estimation in chemometrics The example of multivariate (and multiway) prediction uncertainty Klaas Faber Chemometry

Uncertainty Prediction Across a Range of Scales: From short ......BERNER, J.: UNCERTAINTY PREDICTION ACROSS A RANGE OF SCALES the ensemble mean are computed for horizontal wind and

Visualizing Uncertainty in the Prediction of Academic Risk

Fault Seal Prediction and Uncertainty Estimation of a Water Wet Fault

Uncertainty and Confidence in Climate Change Prediction

Estimating Uncertainty of Streamflow Simulation using ... · uncertainty. The reasonable estimation of the prediction uncertainty, a valuable for decision making to address water

Update on Monte-Carlo tools to assess uncertainty … and Communicating Uncertainty in Human PK and Dose Prediction ... • Translation uncertainty reflects ... administered or the

Advanced Methods for Determining Prediction Uncertainty in

Estimation uncertainty of direct monetary flood damage to

Evaluating Prediction Uncertainty

The oceans, uncertainty, and climate prediction

UNITE: Uncertainty-based Health Risk Prediction Leveraging

Prediction under Uncertainty in Sparse Spectrum Gaussian Processes with Applications ...bboots/files/rldm_ssgp.pdf · 2021. 8. 10. · Prediction under Uncertainty in Sparse Spectrum

Imagining Replications: Graphical Prediction & Discrete ...Imagining Replications: Graphical Prediction & Discrete Visualizations Improve Recall & Estimation of Effect Uncertainty

Approximating Prediction Uncertainty for Random Forest ... · Approximating Prediction Uncertainty for Random Forest Regression Models John W. Coulston , Christine E. Blinn , Valerie

Uncertainty in prediction and simulation of flow in sewer systems · Uncertainty in prediction and simulation of flow in sewer systems Anders Breinholt PhD Thesis May 2012 DTU Environment

Online Prediction under Model Uncertainty Via Dynamic Model

UNCERTAINTY ANALYSIS OF LONG TERM WIND SPEED PREDICTION ALEX KAPETANOVIC

Compact Representation of Uncertainty in Clusteringpapers.nips.cc/paper/...uncertainty-in-clustering.pdf · For many classic structured prediction problems, probability distributions

Quantification of Uncertainty in Mineral Prospectivity ...fs.unm.edu/QuantificationOfUncertaintyInMineral.pdf · Quantification of Uncertainty in Mineral Prospectivity Prediction

Foreign Direct Investment and Uncertainty: Implications

The Impact of Trajectory Prediction Uncertainty on

PORE PRESSURE PREDICTION AND DIRECT HYDROCARBON …

Ocean Model Uncertainty in Climate Prediction

Using Prediction Uncertainty Analysis to Design Hydrologic ... · Using Prediction Uncertainty Analysis to Design Hydrologic Monitoring Networks: Example Applications from the Great

Quantification of Uncertainty in Mineral Prospectivity Prediction Using Neural Network Ensembles

Estimating Prediction Uncertainty from Geographical

Uncertainty-aware Human Motion Prediction arXiv:2107