shallow-deep networks: understanding and mitigating ... · the wasteful effect of overthinking...
TRANSCRIPT
![Page 1: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/1.jpg)
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
Yiğitcan Kaya, Sanghyun Hong, Tudor Dumitraș
University of Maryland, College Park
ICML 2019 - Long Beach, CA
![Page 2: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/2.jpg)
What is overthinking?
We, especially grad students, often think more than needed to solve a problem.
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
![Page 3: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/3.jpg)
What is overthinking?
We, especially grad students, often think more than needed to solve a problem.
i. Wastes our valuable energy (wasteful)
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
![Page 4: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/4.jpg)
What is overthinking?
We, especially grad students, often think more than needed to solve a problem.
i. Wastes our valuable energy (wasteful)
ii. Causes us to make mistakes (destructive)
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
![Page 5: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/5.jpg)
Do deep neural networks overthink too?
Without requiring the full depth, DNNs can correctly classify the majority of samples.
Experiments on four recent CNNs and three common image classification tasks
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
![Page 6: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/6.jpg)
Do deep neural networks overthink too?
Without requiring the full depth, DNNs can correctly classify the majority of samples.
i. Wastes computation for up to 95% of the samples (wasteful)
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
![Page 7: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/7.jpg)
Do deep neural networks overthink too?
Without requiring the full depth, DNNs can correctly classify the majority of samples.
i. Wastes computation for up to 95% of the samples (wasteful)
ii. Occurs in ~50% of all misclassifications (destructive)
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
![Page 8: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/8.jpg)
How do we detect overthinking?
Internal classifiers allow us to observe whether the DNN correctly classifies the sample at an earlier layer.
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
![Page 9: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/9.jpg)
How do we detect overthinking?
Internal classifiers allow us to observe whether the DNN correctly classifies the sample at an earlier layer.
➢Our generic Shallow-Deep Network (SDN) modification introduces internal classifiers to DNNs.
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
![Page 10: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/10.jpg)
The SDN modification
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
conv1 conv2 conv3 conv4 full
FR
full
Input
Internal Prediction
FinalPrediction
Internal Classifier
Internal Layers Final Classifier
Original CNN
SDN modification
Applied to VGG, ResNet, WideResNet and MobileNet.
![Page 11: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/11.jpg)
The SDN modification
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
ChallengeHow to train accurate internal classifiers?
![Page 12: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/12.jpg)
The SDN modification
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
ChallengeHow to train accurate internal classifiers?
Prior WorkClaims this hurts the accuracy in off-the-shelf DNNs
Proposes a unique architecture[1]
[1] Huang, Gao, et al. "Multi-scale dense convolutional networks for efficient prediction." ICLR 2018
![Page 13: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/13.jpg)
The SDN modification
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
ChallengeHow to train accurate internal classifiers?
ResultsOur modification often improves the original
accuracy by up to 10%.(See our poster)
![Page 14: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/14.jpg)
The wasteful effect of overthinking
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
conv1 conv2 conv3 conv4 full
FR
full
Input
Horse✔
Horse✔
Wasteful for the correct classification
![Page 15: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/15.jpg)
The wasteful effect of overthinking
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
ChallengeHow can we know where in the DNN to stop?
![Page 16: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/16.jpg)
The wasteful effect of overthinking
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
ChallengeHow can we know where in the DNN to stop?
Our SolutionClassification confidence of the internal classifiers
![Page 17: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/17.jpg)
The wasteful effect of overthinking
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
Our SolutionClassification confidence of the internal classifiers
ResultsA confidence-based early exit scheme reduces the
average inference cost by up to 50%. (See our poster)
![Page 18: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/18.jpg)
The destructive effect of overthinking
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
conv1 conv2 conv3 conv4 full
FR
full
Input
Horse✔
Dog X
Destructive for the correct classification
![Page 19: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/19.jpg)
The destructive effect causes disagreement
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
conv1 conv2 conv3 conv4 full
FR
full
Input
Horse✔
Dog X
![Page 20: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/20.jpg)
The destructive effect causes disagreement
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
ChallengeHow can we quantify the internal
disagreement?
Our SolutionThe confusion metric
![Page 21: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/21.jpg)
The destructive effect causes disagreement
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
Our SolutionThe confusion metric?
ResultsConfusion indicates whether a misclassification is likely.
Confusion is a reliable error indicator.(See our poster)
![Page 22: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/22.jpg)
The destructive effect causes disagreement
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
Our SolutionThe confusion metric?
ResultsBackdoor attacks [2] also increase the confusion of the
victim DNN for malicious samples.(See our poster)
[2] Gu, Tianyu, et al. "BadNets: Evaluating Backdooring Attacks on Deep Neural Networks." IEEE Access 7 (2019): 47230-47244.
![Page 23: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/23.jpg)
Implications
• Eliminating overthinking would lead to a significant boost in accuracy and inference-time.
• We need DNNs that can adjust their complexity based on the required feature complexity.
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
![Page 24: Shallow-Deep Networks: Understanding and Mitigating ... · The wasteful effect of overthinking Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Challenge How](https://reader030.vdocument.in/reader030/viewer/2022040115/5e7ab9c318276f2b957d9e4f/html5/thumbnails/24.jpg)
Thank you!
Don’t overthink! Come and see our poster!
Pacific Ballroom – Poster #24 – 06:30-09:00 PM
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
For more details, visit our website http://shallowdeep.network