deep learning for (histopathology) medical image …...deep learning in a nutshell tue.nl/image...

Deep learning for (histopathology) medical image analysis

Mitko Veta

Outline

Introduction and previous work

Medical image analysis challenges

Overview of current work in deep learning for medical image analysis (mostly histopathology)

tue.nl/image PAGE 122-11-2018

Introductionand previous work


PhD thesis


UMC Utrecht2010 - 2014

Cancer prognostication


Molecular changesPrognosis Phenotype

ViewTreatment

plan

Pathology lab workflow


Two research paths

Imaging biomarker detection:Develop methods that can seamlessly fit into the pathology lab workflow

Imaging biomarker discovery:Integration with molecular dataFocus on cancer types for which current prognostic tools fail


Nuclei segmentation

Method based on mathematical morphologyCan be used to objectively measure tissue parameters such as the mean nuclei area


M. Veta et al., "Automatic Nuclei Segmentation in H&E Stained Breast Cancer Histopathology Images“, PloS One, 2013

Nuclei segmentation


M. Veta et al., "Prognostic value of automatically extracted nuclear morphometric features in whole slide images of male breast cancer“, Mod Pathol, 2012

Prognostic value for breast cancer patients

Mitosis detection / tumor proliferation


Challengesfriendly competitions in solving medical image analysis problems


Grand challenges in medical image analysis


www.grand-challenge.org

AMIDA13


M. Veta et al., ” Assessment of algorithms for mitosis detection in breast cancer histopathology images“, Med Image Anal, 2015

AMIDA13


“Wow, deep learning works. In a few years everyone will be doing it!” (2013)

Deep learningin a nutshell


“Classical” machine learning


Figure generated with Tensorflow Playground


PAGE 1922-11-2018tue.nl/image



PAGE 2022-11-2018

feature engineering

tue.nl/image


A simple neural network

PAGE 2122-11-2018

learned feature representation

tue.nl/image


Deep learning

PAGE 2222-11-2018

many hidden layers = “deep”

tue.nl/image


Deep learning

Deep learning is the return of neural networks to the mainstream

Technological changes that made this possible:- Availability of large training datasets- Use of GPUs for general purpose computing


Image from The Economist

Convolutional neural networks

Make use of the structure of images to reduce the number of parameters (but keep the representational power)


convolutional layer with one feature map

“typical” deep convolutional neural network

Example: detection of mitotic figures


M Veta et al. “Mitosis counting in breast cancer: Object-level interobserver agreement and comparison to an automatic method”, PloSOne, 2016

How does it compare to pathologists?


M Veta et al. “Mitosis counting in breast cancer: Object-level interobserver agreement and comparison to an automatic method”, PloSOne, 2016

Example: nuclei detection and measurement


M Veta et al. “Cutting out the middleman: measuring nuclear area in histopathology slides without segmentation”, MICCAI, 2016

TUPAC16


tupac.tue-image.nl

TUPAC16


Main task:Produce a slide-level tumor proliferation scoreTwo proliferation scores: by pathologists and by molecular analysis

TUPAC16


TUPAC16

Prediction of molecular proliferation score from image dataAverage of all automatic methods: r = 0.696


CAMELYON16


B.E. Bejnordi, M. Veta et al., “Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer”, JAMA, 2017

CAMELYON16


Future of medical image analysis challenges

Increase the quality of the challenge datasets so the results can be used as evidence for regulatory approvalof algorithms

Challenge datasets as FDA-approved Medical Device Development Tools (MDDTs)


www.fda.gov/MedicalDevices/ScienceandResearch/MedicalDeviceDevelopmentToolsMDDT/

Current workin deep learning for histopathology image analysis


Obstacles to clinical translation

Tissue appearance variability

Large image size e.g. 1 slide ~ 50,000×50,000 pixels

Lack of ready-to-use image annotations




How to best address the appearance variability?Necessary in order to apply the methods across many different pathology labs.Currently addressed with ad-hoc approaches such as staining normalization. Can we find a more systematic solution in the context of deep learning?



2D-embedding of CNN features

2-slide dataset (4x40 patches):

...

...

...

...

Domain A

Domain B

Deep learning models can learn representations that contain irrelevant information

M. Lafarge et al., “Domain-adversarial neural networks to address the appearance variability of histopathology images”, MICCAI DLMIA, 2018

Domain-adversarial neural networks


Domain discriminator extensionLearns to identify the slide of origin from the extracted features.

CONV4x4

ReLU

MP

CONV3x3

ReLU

MP

Mitosis probability

Domain probability

63x63x3INPUT

CONV3x3

ReLU

CONV3x3

ReLU

CONV3x3

ReLU

FC

ReLU

FC

softmax

CONV3x3

ReLU

FC

ReLU

FC

softmax

MP MP MP MP

Explicitly learn features that are independent of the domain of the training samples


Domain-adversarial neural networks


Explicitly learn features that are independent of the domain of the training samples


Baseline Stain normalization Color augmentation DANN

Domain invariance in the model representation.

Large image size

Test-time data augmentation (e.g. by exploring rotational invariance) often leads to improved performance, but is computationally expensive for large whole-slide images


Roto-translation covariant CNNs

Framework for rotation and translation covariant deep learning using SE(2) group convolutions


E. Bekkers and M. Lafarge et al., “Roto-translation covariant convolutional networks for medical image analysis”, MICCAI, 2018

Lack of ready-to-use image annotations

Annotating medical images is a tedious and expensive task as it relies on medical expertsHowever, “weak” image labels can be extracted from routine medical reports or from patient follow-up


Weakly-labelled learning for histopathology image analysis

Goal: DCIS grading, BC survival prediction BC risk prediction

Project in collaboration with UMC Utrecht, Beth Israel Deaconess MC

Datasets: 600+ BC patients with survival, DCIS cases from the UMC Utrecht digital pathology archive, Nurses’ Health Study


Weakly-labelled learning for retina image analysis

Goal: predict diabetes and other systemic diseases from retina images

Project in collaboration with Maastricht UMC, The Maastricht Study


Weakly-labelled learning for retina image analysis

Global labels can still detect “local” image features


Local predictions of model trained with global vessel curvature labels

deep learning for (histopathology) medical image …...deep learning in a nutshell tue.nl/image...

Documents