4/6/18
1
Learning Normalized Inputs for Iterative Estimation
in Medical Image Segmentation
Michal Drozdzal, Gabriel Chartrand, Eugene Vorontsov, Mahsa Shakeri, Lisa Di Jorio, An Tang, Adriana Romero,
Yoshua Bengio, Chris Pal, Samuel Kadoury
Medical imaging modalities - basicsEndoscopyElectron
MicroscopyComputed
TomographyMagnetic Resonance
Imaging
2D/3D
Temporaldimension
Signal scale
2D/3D 3D 3D 2D
No No No Yes
Grayscale RGBHounsfieldscale -
We will t
reat all
data a
s 2D d
ata.
4/6/18
2
Image segmentation
Electron microscopy
CT
Endoscopy
Medical imaging segmentation pipeline
Pre-processing Model Post-processing
4/6/18
3
Medical imaging segmentation pipeline
Pre-processing Model Post-processing
Modality & model specific3D for 2D model
FP reduction
Morphological operations2D/3D CRF
What is used?
Medical imaging segmentation pipeline
Pre-processing Model Post-processing
Fully Convolutional NetworkFCN8
UNET
Jon Long et. al. CVPR 2015Olaf Ronneberger et. al. MICCAI 2015
Modality & model specific3D for 2D model
FP reduction
Morphological operations2D/3D CRF
What is used?
4/6/18
4
Fully Convolutional Networks
Down sampling Up sampling Skip- Pooling- Strided convolutions
- Repeat + convolution- Transposed convolutions
- Concatenate- Sum
This can be ResNet, DenseNet, …This can be ResNet, DenseNet, …
Medical imaging segmentation pipeline
Pre-processing Model Post-processing
Modality specific(handcrafted)
Fully Convolutional NetworkFCN8
UNET
Jon Long et. al. CVPR 2015Olaf Ronneberger et. al. MICCAI 2015
Range normalizationValue clipping
StandardizationN4 (MRI)
Modality & model specific3D for 2D model
FP reduction
Morphological operations2D/3D CRF
What is used?
4/6/18
5
Examples of segmentation pipelines
Range normalizationValue clipping
StandardizationN4 (MRI)
Histogram equalizationGaussian smoothing
2D Unet2D FCN8
2D FC-ResNet3D Unet3D FCN8
3D FC-ResNet
2D CRF3D CRF
Morphological operations
Tools of medical imaging segmentation practitioner:Pre-processing Model Post-processing
Lung segmentation in CT:Lung segmentation in MRI:Liver segmentation in CT:
Standardization + 2DUnet N4 + 2DUNetValue clipping + 3DUnet + morphological operations
Examples of (hypothetical) segmentation pipelines:
Let’s design a model that can be trained with any imaging modality and does not require any pre-processing.
4/6/18
6
Let’s use ResNets
F(x)
x
F(x) + x+
F(x)
x
F(x) + convolution(x)+
convolution
*N *N
x has high impact on feature maps due to skip connections
ResNet uses initial convolution that can adapt input
Recent findings suggest that F() is a transformation close
to identity [1]
We found that FC-ResNetsare more susceptible to data
pre-processing than FCNs
[1] Veit at al. Residual Networks are Exponential Ensembles of Relatively Shallow Networks
Observations about ResNets:
Fully Convolutional
Network
Fully Convolutional
Residual Network
EM
CT
MRI
CT
MRI
EM
What do we propose?
4/6/18
7
Model
Fully Convolutional
Network
Fully Convolutional
Residual Network
Input[1xaxb]
Segmentation map[1xaxb]
Feature map[1xaxb]
+/- 1M parameters +/- 11M parameters
DataElectron Microscopy Computed Tomography Magnetic Resonance
Cell segmentation Lesion segmentation Prostate segmentationN=
30 training images30 testing images
N=105 training volumes30 testing volumes
N=50 training volumes30 testing volumes
4/6/18
8
Experimental setup
Data preparation:> No normalization!> Data augmentation
- Flips- Rotation- Shearing- Elastic transformations- Cropping
Optimization:> RMSprop> Weight decay
EM data results (as of mid 2017)
96.8 96
.9 97 97.1 97
.2
97.2 97
.3
97.7 97
.8 98 98.1
VRAND
Pyramid-LSTM FC-ResNet optree-idsa SCImotif IDSIA Unet CUMedVisionFusionNet IAL Ours
Comparison to published methods Qualitative results(test set)
4/6/18
9
CT data results (as of mid 2017)53
.5 57 61.7 71
.1
D ICE
FCN8 Unet FC-ResNet Ours
Comparison to standard FCNs
Image True segmentation Result
Qualitative results(test set)
MRI data results (as of mid 2017)
79.92 83
.02
74.17
82.39
86.65
2D FCN 3D FCN
Situs OursSRIBHME CAMP-TUM2CUMED
Comparison to published methodsQualitative results
(test set)
4/6/18
10
Fully Convolutional
Network
Fully Convolutional
Residual Network
EM
CT
MRI
CT
MRI
EM
EM
CT
MRI
EM
CT
MRI
Inpu
t int
ensit
y hist
ogra
m:
Norm
alize
d int
ensit
y hist
ogra
m:
Pre-processor effect
Can we quantify the pre-processing effect?
Pre-processor quantification
d( , ) > d( , ) d( , ) = d( , )
Mean Jensen–Shannon distance on validation set
2.99
3.57
5.71
2.98 3.3
5 3.89
2.48 2.
87
3.38
EM CT MRI
input data standardization pre-processor
d(0,0) d(1,0) d(2,0)
d(2,0)
d(1,0) d(1,1)
d(2,1) d(2,2)
d(2,1)
4/6/18
11
Wrap up
A low capacity FCN can serve as a learnable pre-processor.
Combining learnable pre-processor with FC-ResNet yields very good results on a variety of image modalities.
Single pipeline for all type of medical data!
No need to handcraft data pre-processing.
Want to work at the confluence of academia and industry?
MILA has open positions for:• Professors
• Software engineers• Director of software
• R&D & technology transfer• Linux sysadmins
is hiring!
https://tinyurl.com/mila-jobs
4/6/18
12
Learning Normalized Inputs for Iterative Estimation
in Medical Image Segmentation
Michal Drozdzal, Gabriel Chartrand, Eugene Vorontsov, Mahsa Shakeri, Lisa Di Jorio, An Tang, Adriana Romero,
Yoshua Bengio, Chris Pal, Samuel Kadoury