machine learning for bone scans(to serve as a proof of concept) our algorithm a convolutional neural...
TRANSCRIPT
![Page 1: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/1.jpg)
Machine Learning for Bone Scans
Benjamin FangDepartment of RadiologyQueen Mary Hospital
![Page 2: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/2.jpg)
Preamble
• Demand for medical imaging ever increasing• Widening service gap• Not enough radiologists• Need new tools to help => AI
![Page 3: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/3.jpg)
![Page 4: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/4.jpg)
![Page 5: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/5.jpg)
Russakovsky et al., 2015
Error rate (lower is better)
Computer surpassed human
Image source: https://recruitingdaily.com/recruiting‐grudge‐match‐wins‐humans‐vs‐machines/
![Page 6: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/6.jpg)
ObjectiveDevelop a computer algorithm that can recognize bone metastasis in a bone scan image.(To serve as a proof of concept)
![Page 7: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/7.jpg)
Our AlgorithmA Convolutional Neural Network
Input layer:512x512x1
10x10x1x32Convolution RELU 4x4 Maxpool
10x10x32x64Convolution RELU 4x4 Maxpool
8x8x64x64Convolution RELU 2x2 Maxpool
8x8x64x64Convolution RELU 2x2 Maxpool 200‐neuron
Fully connected layer
2‐neuronFully connected layer Softmax
4 Convolutional layers 2 Fully connected layers
Total number of learnable parameters: 3200+204800+524288+51200+400+32+64+64+64+200+2=854,586
Output:Probability of Metastasis (treated as positive if >50%)
![Page 8: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/8.jpg)
Convolutional Neural Network (CNN)
• What is it?– A type of artificial neural network– Uses convolutions (a process whereby featuresare extracted from an image)
![Page 9: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/9.jpg)
Artificial neural network
RELU activation function
http://cs231n.stanford.edu/
![Page 10: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/10.jpg)
Neurons in layers
Image source: MathWorkImage source: https://theconversation.com/deep‐learning‐and‐neural‐networks‐77259
![Page 11: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/11.jpg)
Training our algorithm
1. Forward pass: Feed the network a batch of bone scan images and calculate the predictions
2. Loss: Quantify how good the predictions are3. Back propagation: Calculate how each parameter of the network affects the
predictions4. Optimization: Change each parameter a little to improve the predictions5. Iterate: back to step 1
![Page 12: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/12.jpg)
Our dataset
• 106 Bone scan images58: Metastasis present48: No metastasis
• Diagnosis (ground truth) decided by a Nuclear Medicine Specialist with 29 Years experience in bone scan interpretation
![Page 13: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/13.jpg)
Our dataset
1024x1024 pixelsSingle channel (8‐bit grey scale)
No patient demographic infoNo medical history
Anterior whole body scan Posterior whole body scan
Label: ‐Metastasis present or absent
![Page 14: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/14.jpg)
Image Augmentation(Create infinite variations of our images)
Rotation (‐10 to +10 degrees)Translation (‐45 to +45 pixels)Zooming (‐40 to +40 pixels)Occlusion (left, right or none)
![Page 15: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/15.jpg)
Division of data for training/validation/testing
1/3 (35) assigned for testing2/3 (71) assigned for training/validation
Subdivided into 3 parts
Validation
Training Testing
3‐fold cross validation
Final training
106 Bone scans
![Page 16: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/16.jpg)
Training / Validation
Accuracy
Training step
‐ Training ‐‐ Validation ‐
![Page 17: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/17.jpg)
Final test resultsAccuracy: correct/(correct + incorrect) = 33/(33 +2) = 94%Sensitivity: TP /(TP + FN) = 19 / (19 +1) = 95%Specificity: TN /(TN + FP) = 14 / (14 + 1) = 93%AUC (ROC) = 0.94
![Page 18: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/18.jpg)
What did it get wrong?
![Page 19: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/19.jpg)
![Page 20: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/20.jpg)
Conclusion
• CNNs are very powerful• Our bone scan CNN performed very well despite trained on only a small number of images.
What if our training dataset was much larger?Human expert level performance or better probably achievable.
![Page 21: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/21.jpg)
Some other areas where CNN can be applied
Plain radiographCTMRIUSGMicroscopy images: Pathology / microbiology Clinical photos: dermatologyCapsule endoscopy………etc.
Areas involving visual recognition
![Page 22: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/22.jpg)
Thank you
![Page 23: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/23.jpg)
How to utilize these systems
• 2nd read• Screen unreported exams• Exam prioritization
![Page 24: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/24.jpg)
How to build more of these AI algorithms to help us
• Current limiting factor: Lack of well curated, large training datasets
• Way forward:• For future reports: Structured reporting (standard
templates/checklists etc.)• For past reports:
• Data mining algorithms (neural networks again)• Manual data mining unlikely to be feasible.
![Page 25: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/25.jpg)
Loss function
A quantification of how accurate/inaccurate the prediction is.
Cross Entropy:L=−y log(y^)
L : Lossy^ : predictedY : ground truth
![Page 26: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/26.jpg)
Convolution
Image source: http://machinelearninguru.com/computer_vision/basics/convolution/convolution_layer.html
Each value within a kernel is a learnable parameter
![Page 27: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/27.jpg)
3‐fold Training‐Validation
Accuracy
Training steps
![Page 28: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/28.jpg)
Loss decreases with training
Epoch = number of cycles the network has gone through the whole dataset
![Page 29: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/29.jpg)
The Digital Image
Image source: https://developer.apple.com/library/content/documentation/Performance/Conceptual/vImage/ConvolutionOperations/ConvolutionOperations.html
![Page 30: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/30.jpg)
What about convolution?
‐ A mathematical method to extract features from an image.
‐ Features are represented in different levels of abstraction
![Page 31: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/31.jpg)
Division of data for training/validation/testing
1/3 (35) assigned for testing2/3 (71) assigned for training/validation
Subdivided into 3 parts
Validation
Training Testing
3‐fold cross validation
Final training
106 Bone scans
![Page 32: Machine Learning for Bone Scans(To serve as a proof of concept) Our Algorithm A Convolutional Neural Network Input layer: ... Anterior whole body scan Posterior whole body scan Label:](https://reader034.vdocument.in/reader034/viewer/2022043001/5f7bd4a84759f13c4c09185b/html5/thumbnails/32.jpg)
Russakovsky et al., 2015
Error rate (lower is better)
CNN propelled computer vision to surpass human
Advent of CCNAll CCNs