model-based image reconstruction using deep learned priors ... · 1 lee et al. deep artifact...
TRANSCRIPT
![Page 1: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/1.jpg)
Model-Based Image Reconstruction using
Deep Learned Priors (MoDL)
Hemant K. Aggarwal, Merry P. Mani, and Mathews Jacob
April 6, 2018
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 1 / 25
![Page 2: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/2.jpg)
Inverse Problems & Classical Solutions
Model-Based Problem Formulation
Given y = Ax + n, A 2 CM⇥N , x 2 CN
x̂ = argminx
ky � Axk22 + �R(x)
R(x): regularization priors
Total Variation
Wavelet-based sparsity
Plug and Play denoisersI BM3D, non-local means
1 Venkatakrishnan et al. Plug-and-Play priors for model based reconstruction GlobalSIP, 2013Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 2 / 25
![Page 3: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/3.jpg)
Alternative: black-box deep learning approaches
Deep Artifact Learning1U-NET architecture2
Joint Learning of Image manifold & Inverse: Challenges
Large network: lots of training data
Sensitive to acquisition setting: image matrix, undersampling patternI Need several trained networks
1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.011202 Jin et al. Deep Convolutional Neural Network for Inverse Problems in Imaging MRI IEEE TIP, 2017
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 3 / 25
![Page 4: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/4.jpg)
MoDL: model based recovery with DL priors
Model-Based Problem Formulation
x = argminx
kAx� bk22| {z }data consistency
+� kNw
(x)k2| {z }regularization
N
w
: Predictor of noise and alias patterns in x
Determine x that is data-consistent and minimize aliasing
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 4 / 25
![Page 5: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/5.jpg)
Denoising using noise predictor Nw
Model-Based Problem Formulation
x = argminx
kAx� bk22| {z }data consistency
+� kNw
(x)k2| {z }regularization
Denoiser
Dw
(x) = (I �Nw) (x) = x�Nw
(x).
ConvBN
ReLUN-times
ConvBN
ReLU
ConvBN
Layer 1 Layer 2Layer N
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 5 / 25
![Page 6: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/6.jpg)
Alternating minimization
Problem Formulation
x = argminx
kAx� bk22 + � kx�Dw
(x)k22
Algorithm
zk = Dw
(xk)
xk+1 =⇣A
HA+ � I
⌘�1 ⇣A
Hb+ � zk
⌘
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 6 / 25
![Page 7: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/7.jpg)
Recursive MoDL Architecture
zk = Dw
(xk)
xk+1 =⇣A
HA+ � I
⌘�1 ⇣A
Hb+ � zk
⌘
Iterate
CNN-based Denoiser
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 7 / 25
![Page 8: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/8.jpg)
Training: unroll the recursive network
IterateIteration
K-times
IterationKth1st
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 8 / 25
![Page 9: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/9.jpg)
Shared weights: chain rule for gradients
IterationKthIteration1st Iterationthk
@wC =KX
k=1
@zkC @wzk
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 9 / 25
![Page 10: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/10.jpg)
Joint training is better than pre-trained denoisers
5 10 15 20 25
18
21
24
27
30
33
36
39
42
MoDL iterations during testing
PSNR(dB)
MoDL trained with 1 iterationMoDL trained with 10 iterationsDenoising model: 1 iterationDenoising model: 10 iterations
1 Chang et al. One N/w to Solve Them All: Solving Linear Inverse Prob. using Deep Projection Models, ICCV, 2017Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 10 / 25
![Page 11: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/11.jpg)
Di↵erences with current iterative approaches
Recurrent GAN3Cascade Network4
Challenges
Di↵erent networks at each iteration: not consistent with model based frameworkI Large capacity: Require significantly more training data
More training data available: increase complexity of networks at each iteration
3 Mardani et al. Recurrent GAN for Proximal Learning and Automated Compressive Image Recovery CVPR, 20184 Schlemper et al. A Deep Cascade of CNN for Dynamic MR Image ReconstructionTMI, 20185 Hammernik et al. Learning a Variational Network for Reconstruction of Accelerated MRI Data MRM, 2017
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 11 / 25
![Page 12: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/12.jpg)
MoDL: Can be trained with less training data
50 100 300 600 1,200 2,10021
22
23
24
25
26
27
28
29
30
31
Number of training samples
PSNR(dB)
7L with sharingparameters: 1885553L w/o sharingparameters: 199095
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 12 / 25
![Page 13: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/13.jpg)
MoDL: Weight sharing vs without weight sharing
Original slice A
HB at 6x, 24.97 dB w/o sharing, 32.93 dB with sharing, 38.67 dB
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 13 / 25
![Page 14: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/14.jpg)
MoDL: Insensitivity to acquisition conditions
6-Fold, 39.43 dB 8-Fold, 38.47 dB 10-Fold, 37.75 dB 12-Fold, 36.42 dB 14-Fold, 35.87 dB
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 14 / 25
![Page 15: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/15.jpg)
Parallel MRI: (AHA + �I ) not analytically invertibleSampling Mask
Fourier Transform
Coil Sensitities
= = =
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 15 / 25
![Page 16: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/16.jpg)
Current approaches: Gradient descent (ISTA)
Alternating minimization
zk = Dw
(xk)
xk+1 = argminx
kAx � bk22 + �kx � zk k22
Gradient Descent to minimize DC subproblem5
xk+1 = xk � 2(AHA+ �I )xk + 2AH
b + 2�zk
Shrinkage is cheap in CS setting: fast convergence
Each DC block is in-expensive
6 Hammernik et al. Learning a Variational Network for Reconstruction of Accelerated MRI Data. MRM, 20176 Wang et al. Deep Networks for Image Super-Resolution with Sparse Prior. ICCV, 2015
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 16 / 25
![Page 17: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/17.jpg)
Training di�culties with large unrolled network
Iterate
CNN-based Denoiser
Gradient Descent Step
Large number of iterations
Large network with unrollingI Does not fit on GPUs
Hammernik et al: does not use weight sharing
3 Hammernik et al. Learning a Variational Network for Reconstruction of Accelerated MRI Data MRM, 2017Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 17 / 25
![Page 18: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/18.jpg)
Solution: Numerical Optimization within DL Network
Sub-Problems
zk = Dw
(xk)
xk+1 = argminxkAx � bk22 + �kx � zk k22
Iterate
CNN-based Denoiser
Conjugate Gradient
CG within network
Faster convergence than ISTA
Need fewer iterations: use larger network on GPU
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 18 / 25
![Page 19: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/19.jpg)
Backpropagation through CG Layer
IterationKthIteration1st Iterationthk
Gradient Computation
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 19 / 25
![Page 20: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/20.jpg)
CG within network: improved performance
1 2 3 4 5 6 7 8 9 1028
30
32
34
36
38
Number of iterations
PSNR(dB)at
10-Fold
Gradient DescentConjugate Gradient
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 20 / 25
![Page 21: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/21.jpg)
Parallel Imaging with DL (6x)
Original Image A
HB , 22.93 dB Tikhonov, 34.16 dB
CSTV, 35.20 dB Grad.Desc., 38.29 dB MoDL, 40.33 dB
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 21 / 25
![Page 22: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/22.jpg)
Parallel Imaging with DL (6x)
Tikhonov Compressed Sensing Gradient Descent Proposed MoDL
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 21 / 25
![Page 23: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/23.jpg)
Parallel Imaging with DL (8x)
Original Image A
HB , 23.82 dB Tikhonov, 32.05 dB
CSTV, 34.43 dB Grad.Desc., 35.22 dB MoDL, 37.95 dB
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 22 / 25
![Page 24: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/24.jpg)
Parallel Imaging with DL (8x)
Tikhonov Compressed Sensing Gradient Descent Proposed MoDL
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 22 / 25
![Page 25: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/25.jpg)
MoDL-SToRM: Use patient-specific image priors
MoDL with SToRM prior
x = argminx
kAx� bk22| {z }data consistency
+�1 kNw
(x)k2| {z }regularization
+�2 ktr(XT
WX)k2| {z }SToRM Prior
1 Biswas et al. Model-based Free Breathing Cardiac MRI Recon. using Deep Learned & SToRM priors ICASSP, 2018Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 23 / 25
![Page 26: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/26.jpg)
MoDL-SToRM: Use patient-specific image priors
MoDL with SToRM prior
x = argminx
kAx� bk22| {z }data consistency
+�1 kNw
(x)k2| {z }regularization
+�2 ktr(XT
WX)k2| {z }SToRM Prior
1 Biswas et al. Model-based Free Breathing Cardiac MRI Recon. using Deep Learned & SToRM priors ICASSP, 2018Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 23 / 25
![Page 27: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/27.jpg)
Dynamic image recovery using MoDL-SToRM
SToRM recon from 45s ofacquisition
SToRM recon from 5s ofacquisition
MoDL recon from 5s ofacquisition
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 24 / 25
![Page 28: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/28.jpg)
Conclusion
Integrating DL priors with model based reconstruction: systematic approachI Weight sharing: reduced training dataI More training data: better performance with iterating shared layers
Decouple inversion of forward model from image manifold learningI Relatively insensitive to acquisition settingsI Exploit fast algorithms for forward model evaluation
Numerical optimization blocks within deep networkI Complex forward model: parallel MRII Faster convergence compared to LISTAI Add additional priors: (e.g. subject specific priors)
Model based deep learning image recoveryI Fast image recoveryI Improved image quality
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 25 / 25
![Page 29: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/29.jpg)
Conclusion
Integrating DL priors with model based reconstruction: systematic approachI Weight sharing: reduced training dataI More training data: better performance with iterating shared layers
Decouple inversion of forward model from image manifold learningI Relatively insensitive to acquisition settingsI Exploit fast algorithms for forward model evaluation
Numerical optimization blocks within deep networkI Complex forward model: parallel MRII Faster convergence compared to LISTAI Add additional priors: (e.g. subject specific priors)
Model based deep learning image recoveryI Fast image recoveryI Improved image quality
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 25 / 25
![Page 30: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/30.jpg)
Conclusion
Integrating DL priors with model based reconstruction: systematic approachI Weight sharing: reduced training dataI More training data: better performance with iterating shared layers
Decouple inversion of forward model from image manifold learningI Relatively insensitive to acquisition settingsI Exploit fast algorithms for forward model evaluation
Numerical optimization blocks within deep networkI Complex forward model: parallel MRII Faster convergence compared to LISTAI Add additional priors: (e.g. subject specific priors)
Model based deep learning image recoveryI Fast image recoveryI Improved image quality
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 25 / 25
![Page 31: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/31.jpg)
Conclusion
Integrating DL priors with model based reconstruction: systematic approachI Weight sharing: reduced training dataI More training data: better performance with iterating shared layers
Decouple inversion of forward model from image manifold learningI Relatively insensitive to acquisition settingsI Exploit fast algorithms for forward model evaluation
Numerical optimization blocks within deep networkI Complex forward model: parallel MRII Faster convergence compared to LISTAI Add additional priors: (e.g. subject specific priors)
Model based deep learning image recoveryI Fast image recoveryI Improved image quality
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 25 / 25
![Page 32: Model-Based Image Reconstruction using Deep Learned Priors ... · 1 Lee et al. Deep artifact learning for compressed sensing and parallel MRI arXiv, 1703.01120 2 Jin et al. Deep Convolutional](https://reader036.vdocument.in/reader036/viewer/2022071102/5fdb28d5ad5067652b3a48f9/html5/thumbnails/32.jpg)
Thank You
Full paper: https://arxiv.org/abs/1712.02862
Computational Biomedical Imaging Group Laboratory
http://research.engineering.uiowa.edu/cbig/
Aggarwal, Mani, and Jacob MoDL: Model Based Deep Learning April 6, 2018 25 / 25