![Page 1: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/1.jpg)
Nanyang Assistant Professor
School of Electrical and Electronic Engineering (EEE)Nanyang Technological University (NTU)
Machine Learning for CS MRI:From Model-Based Methods to Deep Learning
Bihan Wen
![Page 2: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/2.jpg)
• Artificial Intelligence (AI): Data-Driven Models
Data-Driven Models > Analytical Models
![Page 3: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/3.jpg)
• Artificial Intelligence (AI): Data-Driven Models
• Machine Learning
Deep Learning, Optimization, Sparse Coding, etc.
![Page 4: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/4.jpg)
• Artificial Intelligence (AI): Data-Driven Models
• Machine Learning
• Solutions with Mathematical Analysis
Mathematical Analysis, Convergence Guarantee, etc.
![Page 5: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/5.jpg)
• Artificial Intelligence (AI): Data-Driven Models
• Machine Learning
• Solutions with Mathematical Analysis
• Applications with State-of-the-art Results
Restoration Analysis
![Page 6: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/6.jpg)
• Video Analysis
• Computational Imaging• Image Restoration
• Image Analysis
Computer Vision vs. Image Reconstruction
Computer Vision
Image Reconstruction
![Page 7: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/7.jpg)
Computer Vision vs. Image Formation
“Cat”
understanding
Sensing
Image Reconstruction
Computer Vision
![Page 8: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/8.jpg)
• Compressed Sensing MRI
• Why Do We Need Data Models?
• Tutorial on Transform Learning (TL) for MRI
• From Model-Based Method to Deep Learning
![Page 9: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/9.jpg)
Magnetic Resonance Imaging (MRI)
MR sampling
MR image reconstruction
![Page 10: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/10.jpg)
CS MRI
MR sampling
MR image reconstruction
Image SpaceK-Space
![Page 11: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/11.jpg)
Why MRI?
• Non-invasive
![Page 12: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/12.jpg)
Why MRI?
• Non-invasive
• Non-ionizing
![Page 13: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/13.jpg)
Why MRI?
• Non-invasive
• Non-ionizing
• Variety of Contrast and Visualization
![Page 14: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/14.jpg)
Why Compressed Sensing (CS)?
• Scan time is too long
![Page 15: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/15.jpg)
• Scan time is too long
• Image Resolution
Why Compressed Sensing (CS)?
![Page 16: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/16.jpg)
CS MRI
MR sampling
𝑥𝑥: MR image
𝑦𝑦: k-Space measurement
![Page 17: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/17.jpg)
CS MRI
MR sampling
MR image reconstruction
![Page 18: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/18.jpg)
CS MRI
Naïve Reconstruction
under-sampling
CS: better reconstruction via image modeling
![Page 19: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/19.jpg)
CS MRI
Compressed Sensing MRI
Transform-domainSparsity
Sparsity as the regularizer
![Page 20: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/20.jpg)
What makes images look like images?
How to distinguish desired pattern from others?
![Page 21: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/21.jpg)
Data Models
![Page 22: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/22.jpg)
Machine Learning
GMM
PCA
DictionaryLearning
TransformLearning
CNN
RNN U-Net
GAN
TransferLearning
FCN
Data Models
Shallow Methods
Deep Methods
![Page 23: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/23.jpg)
![Page 24: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/24.jpg)
Highly sparse DCT coefficients
2D Discrete Cosine Transform (2D DCT)
Natural Image
![Page 25: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/25.jpg)
i.i.d. White Gaussian
2D Discrete Cosine Transform (2D DCT)
i.i.d. White Gaussian Noise
![Page 26: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/26.jpg)
Sparsity and Beyond
[Wen et al., SPM 2020]
![Page 27: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/27.jpg)
Transform Learning for Better Sparsity
Transform-Learning (TL) based regularizer
Fixed Transform
2D DCT Learned Transform[Wen et al., SPM 2020]
![Page 28: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/28.jpg)
TL-based MRI
1. Sparsifying Transform Learning (STL)
2. Unitary Transform Learning (UT)
3. Learning a UNIon of Transforms (UNITE)
4. Flipping and Rotation Invariant Sparsifying Transform (FRIST)
5. Sparsifying TRansfOrm Learning and Low-Rankness (STROLLR)
[Wen et al., SPM 2020]
![Page 29: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/29.jpg)
SparsifyingTransformLearning
Octobos: Learning a Union of Transforms
Online Transform Learning
STROLLR: Transform Learning with Low-Rank Regularization
FRIST: flipping and rotation invariant Transform Learning
[IJCV 15’, IP 17’, JSTSP 15’, TIP 20’]
![Page 30: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/30.jpg)
TL-based MRI
1. Sparsifying Transform Learning (STL)
Well-conditioning Regularizer for 𝑊𝑊
1. Coefficient 𝜆𝜆 controls the condition number
2. Prevents trivial solution, i.e., 𝑊𝑊 = 0
![Page 31: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/31.jpg)
TL-based MRI
2. Unitary Transform Learning (UT)
1. When 𝜆𝜆 → ∞, it is equivalent to unitary condition.
2. Wavelets, DCT, Discrete Fourier Transforms are all unitary.
3. Closed-form solution
![Page 32: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/32.jpg)
TL-based MRI
Patch extrac
tion
Initialization𝑾𝑾Transform Learning Schemes
Output
Reconstructed�𝒙𝒙
Image Update
K-spaceUnder-samplingMR image
𝒙𝒙Input
K-space𝒚𝒚
Sensing and Reconstruction
[Wen et al., SPM 2020]
![Page 33: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/33.jpg)
TL-based MRI
3. Learning a UNIon of Transforms (UNITE)
1. A union of transforms {𝑊𝑊𝑘𝑘} with the membership {𝐶𝐶𝑘𝑘}.
2. Patches with similar textures will be grouped together.
![Page 34: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/34.jpg)
TL-based MRI
4. Flipping and Rotation Invariant Sparsifying Transform (FRIST)
1. Pre-defined operators Φ𝑘𝑘 ; One parent transform 𝑊𝑊.
2. Prevent overfitting; handles rotation and flipping.
![Page 35: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/35.jpg)
TL-based MRI
5. Sparsifying TRansfOrm Learning and Low-Rankness (STROLLR)
Transform Learning
Low-Rank Modelling
![Page 36: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/36.jpg)
TL-based MRI
A unified framework for TL-based MRI
[Wen et al., SPM 2020]
![Page 37: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/37.jpg)
Image Priors - Complementary
Can you combine any priors, and always gain?
[Wen et al., Arxiv 2020]
Combine only the complementary priors / image models
![Page 38: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/38.jpg)
Why Compressed Sensing (CS)?
[Wen et al., SPM 2020]
Combine only the complimentary priors / image models
![Page 39: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/39.jpg)
39
Deep Learning
![Page 40: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/40.jpg)
time1990 20102000
Deep Learning
Deep Learning
![Page 41: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/41.jpg)
Connection to Unrolled Neural Networks
1. Improved Performance
2. Robust to Corruptions
![Page 42: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/42.jpg)
Unrolled Transform Learning for MRI
• Unrolled TL-MRI
[Ravishankar et al., ISBI 2018]
• Multi-Layer Transform Residual Learning
![Page 43: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/43.jpg)
Some Results
[Wen et al., SPM 2020]
![Page 44: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/44.jpg)
Some Results
[Wen et al., SPM 2020]
![Page 45: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/45.jpg)
• Deep Model• Multiple free layers
Conventional Deep Learning• Shallow Model
• Equivalently one free layer
![Page 46: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/46.jpg)
• Deep Model• Multiple free layers
• Supervised • Training corpus needed• Data inefficient
Conventional Deep Learning• Shallow Model
• One free layer
• Unsupervised • No training corpus needed• Data efficient
?
![Page 47: 视觉与学习青年学者研讨会 - Machine Learning for CS MRIvalser.org/webinar/slide/slides/20200415/Valse2020_WBH...2020/04/15 · Nanyang Assistant Professor School of Electrical](https://reader036.vdocument.in/reader036/viewer/2022071410/61056df680a2a4640b5744b3/html5/thumbnails/47.jpg)
• Deep Model• Multiple free layers
• Supervised • Training corpus needed• Data inefficient
• Generic• Little assumption• Almost free model
Conventional Deep Learning• Shallow Model
• One free layer
• Unsupervised • No training corpus needed• Data efficient
• Prior-based• Assumption & Understanding
of the Data• Regularizer & structures of the
Model
?