transfer learning and fine-tuning deep neural networks
TRANSCRIPT
![Page 1: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/1.jpg)
© 2014 Microsoft Corporation. All rights reserved.
Anusua Trivedi, Data ScientistAlgorithm Data Science (ADS)
Transfer Learning and Fine-tuning Deep Neural Networks
![Page 2: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/2.jpg)
2
1. Traditional Machine Learning (ML)2. ML Vs. Deep Learning3. Why Deep Learning for Image Analysis4. Deep Convolutional Neural Network (DCNN)5. Transfer Learning DCNN6. Fine-tuning DCNN7. Recurrent Neural Network (RNN)8. Case Studies
Talk Outline
![Page 3: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/3.jpg)
3
Vision Analytics
Recommenda-tion engines
Advertising analysis
Weather forecasting for business planning
Social network analysis
Legal discovery and document archiving
Pricing analysisFraud detection
Churn analysis
Equipment monitoring
Location-based tracking and services
Personalized Insurance
Machine learning & predictive analytics are core capabilities that help business decisions
What is ML?
![Page 4: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/4.jpg)
4
Traditional ML Vs Deep Learning
Deep learning can automatically learn features in data
Deep learning is largely a "black box" technique, updating learned weights at each layer
Traditional ML requires manual feature extraction/engineering
Feature extraction for unstructured data is very difficult
![Page 5: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/5.jpg)
5
1. Image data requires subject-matter expertise to extract key features
2. Deep learning extracts feature automatically from domain-specific images, without any feature engineering technique
3. This step makes the image analysis process much easier
Why use Deep Learning for Image Analysis?
![Page 6: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/6.jpg)
6
Early Work
1. Fukushima (1980) – Neo-Cognitron
2. LeCun (1989) – Convolutional Neural Networks (CNN)
3. With the advent of GPUs, DCNN popularity grew
4. Most popular – AlexNet (on ImageNet images)
![Page 7: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/7.jpg)
7
1. Train networks with many layers
2. Multiple layers work to build an improved feature space
3. First layer learns 1st order features (e.g. edges)
4. 2nd layer learns higher order features
5. Lastly, final layer features are fed into supervised layer(s)
Deep Neural Network (DNN)
![Page 8: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/8.jpg)
8
Deep Convolutional Neural Network (DCNN)
C layers are convolutions, S layers pool/sample
![Page 9: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/9.jpg)
9
Essential components of DCNN
![Page 10: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/10.jpg)
10
Convolution• Conv layers consist of a
rectangular grid of neurons.
• The weights for this are the same for each neuron in the conv layer.
• The conv layer weights specify the convolution filter.
![Page 11: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/11.jpg)
11
PoolingThe pooling layer takes small rectangular blocks from the convolutional layer and subsamples it to produce a single output from that block
![Page 12: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/12.jpg)
12
DCNN Sample - LeNet
![Page 13: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/13.jpg)
13
Transfer Learning & Fine-tuning DCNN
![Page 14: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/14.jpg)
14
1.Non-symbolic frameworks• The main drawback of imperative frameworks
(like torch, caffe etc. ) is manual optimization. • Most imperative frameworks are not easily modified.
2.Symbolic frameworks• Symbolic frameworks (like Theano, Tensorflow, CNTK,
MXNET etc.) can infer optimization automatically from the dependency graph.
• A symbolic framework can exploit much more memory reuse
Deep Learning Frameworks
![Page 15: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/15.jpg)
15
1. Easy to implement new networks
2. Easy to modify existing networks using Lasagne/Keras
3. Very mature python interface
4. Easy to customize with domain-specific data.
5. Transfer learning and fine tuning in Lasagne/Keras is very easy
Theano
![Page 16: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/16.jpg)
16
1. Here we use labeled fluorescein angiography images of eyes to improve Diabetic Retinopathy (DR) prediction.
2. We use a DCNN to improve DR prediction.
Case Study: Diabetic Retinopathy Prediction
![Page 17: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/17.jpg)
17
GoogleNet
![Page 18: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/18.jpg)
18
ImageNet
![Page 19: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/19.jpg)
19
1. We use an ImageNet pre-trained DCNN
2. We fine-tune that DCNN to transfer generic learned features to DR prediction.
3. Lower layers of the pre-trained DCNN contain generic features that can be used for the DR prediction task.
Transfer Learning & Fine-tuning our DCNN model
![Page 20: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/20.jpg)
20
Diabetic Retinopathy
![Page 21: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/21.jpg)
21
Image Augmentation
![Page 22: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/22.jpg)
22
Transfer Learning DCNN
![Page 23: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/23.jpg)
23
Fine-tuning GoogleNet
![Page 24: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/24.jpg)
24
Diabetic Retinopathy Prediction
![Page 25: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/25.jpg)
25
Prediction Results Comparison
Our DCNN improves DR prediction accuracy compared to state-of-the-art Support Vector Machine approaches
IMAGE CLASSIFICATION MEAN ACCURACY
Our fine-tuned DCNN 0.96
Feature Based SVM 0.82
![Page 26: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/26.jpg)
26
Other Uses of this DCNN Model
![Page 27: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/27.jpg)
27
Re-usability of this DCNN Model
1. We fine-tune ImageNet-trained DCNN for medical image analysis
2. We can fine-tune the same ImageNet-trained DCNN model in a completely different domain, and for a completely different task.
![Page 28: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/28.jpg)
28
1. We use the ImageNet-trained DCNN and learn Apparel Classification with Style (ACS) image features through transfer learning and fine-tuning.
2. Then we use a Long short-term memory (LSTM) Recurrent Neural Network (RNN) on the learned image features for the image caption generation.
Case Study: Fashion Image Caption Generation
![Page 29: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/29.jpg)
29
Image Augmentation
![Page 30: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/30.jpg)
30
Transfer Learning DCNN
![Page 31: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/31.jpg)
31
Recurrent Neural Network (RNN-LSTM)
• Recurrent neural networks (RNN) are networks with loops in them, allowing information to persist.
• Long Short Term Memory (LSTM) networks are a special kind of RNN, capable of learning long-term dependencies.
• Good for state-wise (step-by-step) caption generation task.
![Page 32: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/32.jpg)
32
Deep CNN-RNN
Model
![Page 33: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/33.jpg)
33
ACS Images Caption Generation
![Page 34: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/34.jpg)
34
Caption generation using fine-tuned CNN-RNN model
![Page 35: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/35.jpg)
35
Microsoft Cognitive APIs and BOTs
![Page 36: Transfer Learning and Fine-tuning Deep Neural Networks](https://reader034.vdocument.in/reader034/viewer/2022042706/587186221a28ab2c198b4eef/html5/thumbnails/36.jpg)
THANKS!
Questions?