Machine Learning in Medical Imaging: Learning from Large-scale populations

www.cir.meduniwien.ac.at

Georg Langs

CIR Lab Department of Biomedical Imaging and Image Guided Therapy Medical University of Vienna

CSAIL MIT

contextflow

www.contextflow.com

4 problems to solve

• Predict progression and response • Learn from clinical routine data• Detect meaningful disease patterns• Discover groups in populations

2

www.cir.meduniwien.ac.at

I. Prediction and treatment response

3

Retinal disease

4

Predicting progression and outcome

• Can we predict outcome from available information?• Can we predict course of disease and treatment?• Identify the predictive features

5

Time ?

[Vogl et al. 2015]

Vogl WD, Waldstein SM, Gerendas BS, Simader C, Glodan AM, Podkowinski D, Schmidt-Erfurth U, Langs G. Spatio-Temporal Signatures to Predict Retinal Disease Recurrence. in Advances in Information Processing in Medical Imaging., IPMI 2015;24:152-63. link

Predicting disease progression

• Predict if recurrence occurs• Predict time to recurrence to ensure timely treatment

6

[Vogl et al. 2015]

Vogl WD, Waldstein SM, Gerendas BS, Simader C, Glodan AM, Podkowinski D, Schmidt-Erfurth U, Langs G. Spatio-Temporal Signatures to Predict Retinal Disease Recurrence. in Advances in Information Processing in Medical Imaging., IPMI 2015;24:152-63. link

Challenges

• Learn from existing data: heterogeneous images and rich but largely unstructured textual information

• Weird biases • Link subtle multivariate observations to future

disease progression or treatment response• Discover and verify new categories relevant for

prognosis

7

www.cir.meduniwien.ac.at

II. Learning from clinical routine data

8

www.cir.meduniwien.ac.at

Study data vs. routine data

9

1 month: MR/CT

>4TB

1px = 10MB

www.cir.meduniwien.ac.at

Sampling of the body

10

M. Dorfer, R. Donner, and G. Langs, Constructing an un-biased whole body atlas from clinical imaging data by fragment bundling. in proceedings of MICCAI 2013, 2013, pp. 219–226

www.cir.meduniwien.ac.at

Whole body mapping

11Work by Hofmanninger, Krenn, Holzer [Dorfer et al. 2013]

[Dorfer et al. 2013 ]

M. Dorfer, R. Donner, and G. Langs, Constructing an un-biased whole body atlas from clinical imaging data by fragment bundling. in proceedings of MICCAI 2013, 2013, pp. 219–226

Rich but unstructured information

www.cir.meduniwien.ac.at tinyurl.com/medim2015Georg Langs 12

www.cir.meduniwien.ac.at

II. Detecting disease patterns

13

www.cir.meduniwien.ac.at

Lung pattern classification

• We can learn with minimal supervision

• Transfer models across clinical sites, manufacturers

14MEDICAL UNIVERSITY OF VIENNA

Inject unlabelled data to improve representation

Have a small set of labelled data to train classification

[Schlegl et al. MICCAI-MCV 2014]

T. Schlegl, J. Ofner, G. Langs. Unsupervised pre-training across domains improves lung tissue classification. In Proc. of MICCAI MCV'14, 2014

Re-mapping visual features

15

[Hofmanninger et al. CVPR 2015]

J. Hofmanninger and G. Langs. Mapping visual features to semantic profiles for retrieval in medical imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 457–465, 2015 link

Learn to identify clinical findings

16

Images and reports Computational maps

Algorithm

[Hofmanninger et al. CVPR 2015]

J. Hofmanninger and G. Langs. Mapping visual features to semantic profiles for retrieval in medical imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 457–465, 2015 link

Learn to identify clinical findings

17

Images and reports Computational maps

Expert

[Hofmanninger et al. CVPR 2015]

J. Hofmanninger and G. Langs. Mapping visual features to semantic profiles for retrieval in medical imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 457–465, 2015 link

Beyond annotation

18

[Valentinitsch et al. 2013]

A. Valentinitsch, J. M. Patsch, A. J. Burghardt, T. M. Link, S. Majumdar, L. Fischer, C. Schueller-Weidekamm, H. Resch, F. Kainberger, G. Langs. Computational identification and quantification of trabecular microarchitecture classes by 3D texture analysis-based clustering. in Bone 54(1):133-140, 2013 (link)

7.2010 9.2010 11.2011

Disease pattern

Time

UIP

NSIP/EAA

www.cir.meduniwien.ac.at

Identifying disease paths

19

[Vogl et al. 2014]

Wolf-Dieter Vogl, Helmut Prosch, Christina Mueller-Mang, Ursula Schmidt-Erfurth, and Georg Langs. Longitudinal Alignment of Disease Progression in Fibrosing Interstitial Lung Disease. In Proc. MICCAI'14, 2014 link

From categories to relationships

20

query

Search engine: find related cases

21

• KHRESMOI (FP7)- building large scale search engines for medical data

• Resulted in spin-off further developing this search engine: contextflow

www.contextflow.com

www.cir.meduniwien.ac.at

IV. Unsupervised learning to understand populations

22

Identify patient groups

• Chest CTs

• t-SNE embedding based on visual features on the right

• Find structure in a population

• Collaboration with TEAMPLAY

23

[Hofmanninger et al. 2016 MICCAI]

J. Hofmanninger, M. Krenn, M. Holzer, T. Schlegl, H. Prosch, G. Langs Unsupervised Identification of Clinically Relevant Clusters in Routine Imaging Data. in Proceedings of MICCAI 2016.

Can we find meaningful structure?

24

[Hofmanninger et al. 2016 MICCAI]

Terms in reports

J. Hofmanninger, M. Krenn, M. Holzer, T. Schlegl, H. Prosch, G. Langs Unsupervised Identification of Clinically Relevant Clusters in Routine Imaging Data. in Proceedings of MICCAI 2016.

Clusters correspond to findings

25

Zyste, P: <0.001, OddsR: 2.6 Lymphom, P: <0.001, OddsR: 4.2Läsion, P: <0.001, OddsR: 2.1

Erguss, P: <0.001, OddsR: 4.2Pneumothorax, P: <0.001, OddsR: 4.8Atelektase, P: <0.001, OddsR: 3.1

Pneumonie, P: <0.001, OddsR: 7.6Erguss, P: <0.001, OddsR: 5.4Stauung, P: <0.001, OddsR: 5.31

Zyste, P: <0.001, OddsR: 2.6 Lymphom, P: <0.001, OddsR: 4.2Läsion, P: <0.001, OddsR: 2.1

Erguss, P: <0.001, OddsR: 4.2Pneumothorax, P: <0.001, OddsR: 4.8Atelektase, P: <0.001, OddsR: 3.1

Pneumonie, P: <0.001, OddsR: 7.6Erguss, P: <0.001, OddsR: 5.4Stauung, P: <0.001, OddsR: 5.31

Zyste, P: <0.001, OddsR: 2.6 Lymphom, P: <0.001, OddsR: 4.2Läsion, P: <0.001, OddsR: 2.1

Erguss, P: <0.001, OddsR: 4.2Pneumothorax, P: <0.001, OddsR: 4.8Atelektase, P: <0.001, OddsR: 3.1

Pneumonie, P: <0.001, OddsR: 7.6Erguss, P: <0.001, OddsR: 5.4Stauung, P: <0.001, OddsR: 5.31

Cluster

Findings in corresponding

reports

Clustering based on visual data Evaluation based on reported findings

[Hofmanninger et al. 2016 MICCAI]

J. Hofmanninger, M. Krenn, M. Holzer, T. Schlegl, H. Prosch, G. Langs Unsupervised Identification of Clinically Relevant Clusters in Routine Imaging Data. in Proceedings of MICCAI 2016.

Conclusion

• Machine learning enables the use of large-scale data to guide feature construction

• Resulting in powerful classificiation-, regression-, and prediction models

• Identification of predictive markers and novel categories in data

• Key: finding marker patterns in heterogeneous very large-scale imaging data

26

www.cir.meduniwien.ac.atwww.contextflow.com

www.VISCERAL.eu

27

www.cir.meduniwien.ac.at