Machine Learning in Medical Imaging: Learning from Large-scale populations
www.cir.meduniwien.ac.at
Georg Langs
CIR Lab Department of Biomedical Imaging and Image Guided Therapy Medical University of Vienna
CSAIL MIT
contextflow
www.contextflow.com
4 problems to solve
• Predict progression and response • Learn from clinical routine data• Detect meaningful disease patterns• Discover groups in populations
2
Predicting progression and outcome
• Can we predict outcome from available information?• Can we predict course of disease and treatment?• Identify the predictive features
5
Time ?
[Vogl et al. 2015]
Vogl WD, Waldstein SM, Gerendas BS, Simader C, Glodan AM, Podkowinski D, Schmidt-Erfurth U, Langs G. Spatio-Temporal Signatures to Predict Retinal Disease Recurrence. in Advances in Information Processing in Medical Imaging., IPMI 2015;24:152-63. link
Predicting disease progression
• Predict if recurrence occurs• Predict time to recurrence to ensure timely treatment
6
[Vogl et al. 2015]
Vogl WD, Waldstein SM, Gerendas BS, Simader C, Glodan AM, Podkowinski D, Schmidt-Erfurth U, Langs G. Spatio-Temporal Signatures to Predict Retinal Disease Recurrence. in Advances in Information Processing in Medical Imaging., IPMI 2015;24:152-63. link
Challenges
• Learn from existing data: heterogeneous images and rich but largely unstructured textual information
• Weird biases • Link subtle multivariate observations to future
disease progression or treatment response• Discover and verify new categories relevant for
prognosis
7
www.cir.meduniwien.ac.at
Study data vs. routine data
9
1 month: MR/CT
>4TB
1px = 10MB
www.cir.meduniwien.ac.at
Sampling of the body
10
M. Dorfer, R. Donner, and G. Langs, Constructing an un-biased whole body atlas from clinical imaging data by fragment bundling. in proceedings of MICCAI 2013, 2013, pp. 219–226
www.cir.meduniwien.ac.at
Whole body mapping
11Work by Hofmanninger, Krenn, Holzer [Dorfer et al. 2013]
[Dorfer et al. 2013 ]
M. Dorfer, R. Donner, and G. Langs, Constructing an un-biased whole body atlas from clinical imaging data by fragment bundling. in proceedings of MICCAI 2013, 2013, pp. 219–226
Rich but unstructured information
www.cir.meduniwien.ac.at tinyurl.com/medim2015Georg Langs 12
www.cir.meduniwien.ac.at
Lung pattern classification
• We can learn with minimal supervision
• Transfer models across clinical sites, manufacturers
14MEDICAL UNIVERSITY OF VIENNA
Inject unlabelled data to improve representation
Have a small set of labelled data to train classification
[Schlegl et al. MICCAI-MCV 2014]
T. Schlegl, J. Ofner, G. Langs. Unsupervised pre-training across domains improves lung tissue classification. In Proc. of MICCAI MCV'14, 2014
Re-mapping visual features
15
[Hofmanninger et al. CVPR 2015]
J. Hofmanninger and G. Langs. Mapping visual features to semantic profiles for retrieval in medical imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 457–465, 2015 link
Learn to identify clinical findings
16
Images and reports Computational maps
Algorithm
[Hofmanninger et al. CVPR 2015]
J. Hofmanninger and G. Langs. Mapping visual features to semantic profiles for retrieval in medical imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 457–465, 2015 link
Learn to identify clinical findings
17
Images and reports Computational maps
Expert
[Hofmanninger et al. CVPR 2015]
J. Hofmanninger and G. Langs. Mapping visual features to semantic profiles for retrieval in medical imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 457–465, 2015 link
Beyond annotation
18
[Valentinitsch et al. 2013]
A. Valentinitsch, J. M. Patsch, A. J. Burghardt, T. M. Link, S. Majumdar, L. Fischer, C. Schueller-Weidekamm, H. Resch, F. Kainberger, G. Langs. Computational identification and quantification of trabecular microarchitecture classes by 3D texture analysis-based clustering. in Bone 54(1):133-140, 2013 (link)
7.2010 9.2010 11.2011
Disease pattern
Time
UIP
NSIP/EAA
www.cir.meduniwien.ac.at
Identifying disease paths
19
[Vogl et al. 2014]
Wolf-Dieter Vogl, Helmut Prosch, Christina Mueller-Mang, Ursula Schmidt-Erfurth, and Georg Langs. Longitudinal Alignment of Disease Progression in Fibrosing Interstitial Lung Disease. In Proc. MICCAI'14, 2014 link
Search engine: find related cases
21
• KHRESMOI (FP7)- building large scale search engines for medical data
• Resulted in spin-off further developing this search engine: contextflow
www.contextflow.com
www.cir.meduniwien.ac.at
IV. Unsupervised learning to understand populations
22
Identify patient groups
• Chest CTs
• t-SNE embedding based on visual features on the right
• Find structure in a population
• Collaboration with TEAMPLAY
23
[Hofmanninger et al. 2016 MICCAI]
J. Hofmanninger, M. Krenn, M. Holzer, T. Schlegl, H. Prosch, G. Langs Unsupervised Identification of Clinically Relevant Clusters in Routine Imaging Data. in Proceedings of MICCAI 2016.
Can we find meaningful structure?
24
[Hofmanninger et al. 2016 MICCAI]
Terms in reports
J. Hofmanninger, M. Krenn, M. Holzer, T. Schlegl, H. Prosch, G. Langs Unsupervised Identification of Clinically Relevant Clusters in Routine Imaging Data. in Proceedings of MICCAI 2016.
Clusters correspond to findings
25
Zyste, P: <0.001, OddsR: 2.6 Lymphom, P: <0.001, OddsR: 4.2Läsion, P: <0.001, OddsR: 2.1
Erguss, P: <0.001, OddsR: 4.2Pneumothorax, P: <0.001, OddsR: 4.8Atelektase, P: <0.001, OddsR: 3.1
Pneumonie, P: <0.001, OddsR: 7.6Erguss, P: <0.001, OddsR: 5.4Stauung, P: <0.001, OddsR: 5.31
Zyste, P: <0.001, OddsR: 2.6 Lymphom, P: <0.001, OddsR: 4.2Läsion, P: <0.001, OddsR: 2.1
Erguss, P: <0.001, OddsR: 4.2Pneumothorax, P: <0.001, OddsR: 4.8Atelektase, P: <0.001, OddsR: 3.1
Pneumonie, P: <0.001, OddsR: 7.6Erguss, P: <0.001, OddsR: 5.4Stauung, P: <0.001, OddsR: 5.31
Zyste, P: <0.001, OddsR: 2.6 Lymphom, P: <0.001, OddsR: 4.2Läsion, P: <0.001, OddsR: 2.1
Erguss, P: <0.001, OddsR: 4.2Pneumothorax, P: <0.001, OddsR: 4.8Atelektase, P: <0.001, OddsR: 3.1
Pneumonie, P: <0.001, OddsR: 7.6Erguss, P: <0.001, OddsR: 5.4Stauung, P: <0.001, OddsR: 5.31
Cluster
Findings in corresponding
reports
Clustering based on visual data Evaluation based on reported findings
[Hofmanninger et al. 2016 MICCAI]
J. Hofmanninger, M. Krenn, M. Holzer, T. Schlegl, H. Prosch, G. Langs Unsupervised Identification of Clinically Relevant Clusters in Routine Imaging Data. in Proceedings of MICCAI 2016.
Conclusion
• Machine learning enables the use of large-scale data to guide feature construction
• Resulting in powerful classificiation-, regression-, and prediction models
• Identification of predictive markers and novel categories in data
• Key: finding marker patterns in heterogeneous very large-scale imaging data
26
www.cir.meduniwien.ac.atwww.contextflow.com