![Page 1: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/1.jpg)
Image ProcessingFraunhofer
Heinrich Hertz Institute
Workshop “Impact of AI on ICT Infrastructures” 25th April, Xi’an, China
Efficient, Distributed and Interpretable Deep Learning
Dr. Wojciech Samek
Fraunhofer HHI, Machine Learning Group
![Page 2: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/2.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Today’s AI systems
Today’s AI has “superhuman” performance
Most success in image & nlp domain
Key ingredients for the success:
- Huge amounts of training data
- Very deep (black-box) models
- Incredible computing power
![Page 3: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/3.jpg)
Can we also expect such a revolution in ICT ?
Yes, but …
![Page 4: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/4.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
ICT settings are slightly different
Key ingredients for the success:
- Huge amounts of training data
- Very deep (black-box) models
- Incredible computing power
data often distributed
using black-boxes not an option
not available (e.g. mobile devices)
—> distributed learning
—> interpretable learning
—> efficient learning
![Page 5: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/5.jpg)
Efficient Deep Learning
![Page 6: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/6.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
DNNs are large and energy hungry
DNN with Millions of weight parameters
- large size
- energy-hungry training & inference
- many floating point operations
For instance, VGG16
- 16 weight layers
- 138 000 000 parameters
- 553 MB (uncompressed)
- 30940 M float operations (sum+mult) for inference
—> 71 mJ just for the float operations on 45nm CMOS process
(Source: http://www.cs.toronto.edu/~frossard/post/vgg16/)
![Page 7: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/7.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
DNNs are large and energy hungry
1. Design optimized hardware
2. Reduce the complexity of the DNN
Popular research topic …
What can we do to bring deep learning to ICT ?
![Page 8: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/8.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Reducing the complexity of DNNs
1. Network Pruning 2. Weight Quantization
3. Efficient EncodingSparse data format
- reduces storage
- fast multiplications
![Page 9: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/9.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
But are compressed DNNs really sparse ?
Quantization leads to low entropy weight
matrices with weight sharing property.
For such matrices, sparse formats may not
be the most efficient ones.
Weight sharing property: Subsets of connections share the same weight value.
![Page 10: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/10.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
more efficient encoding
of low entropy matrices
New efficient format for compressed DNNs
Compression + Our format
size: 12.8 MB, acc: 68.83 %,
ops: 7225 M, energy: 16 mJ
Compression + sparse format
size: 17.8 MB, acc: 68.83 %,
ops: 10081 M, energy: 22 mJ
VGG-16
size: 553 MB, acc: 68.73 %,
ops: 30940 M, energy: 71 mJ
![Page 11: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/11.jpg)
Distributed Deep Learning
![Page 12: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/12.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Distributed Training
AI Server
Mobile
Client
Mobile
Client
Mobile
Client
Mobile
Client
Our goal
- train a model without sending
client data to the server
- minimize communication
overhead
![Page 13: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/13.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Distributed Training
Training algorithm
1. Initialize all clients with the same W
2. Compute weight updates ΔW locally
and send them to the server
3. Update W and send it to the clients
iterate
It even works if gradient is highly sparsified (99.9 %)
(see Lin et al. 2018)
We have very promising extension of this approach.
![Page 14: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/14.jpg)
Interpretable Deep Learning
![Page 15: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/15.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Can we trust these black boxes ?
verify
systemunderstand
weaknesseslegal
aspects learn new
strategies
![Page 16: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/16.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Can we trust these black boxes ?
![Page 17: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/17.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Can we trust these black boxes ?
![Page 18: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/18.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Opening the black box
(Lapuschkin et al., 2016)
![Page 19: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/19.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Upcoming tutorials on interpretability
![Page 20: Efficient, Distributed and Interpretable Deep Learning€¦ · Can we also expect such a revolution in ICT ? Yes, but … Wojciech Samek: Efficient, Distributed and Interpretable](https://reader033.vdocument.in/reader033/viewer/2022042223/5ec9707438000a687d6ec33a/html5/thumbnails/20.jpg)
Wojciech Samek: Efficient, Distributed and Interpretable Deep Learning
Thank you for your attention
Questions ???
Contact Information:
Wojciech Samek
Fraunhofer HHI, Machine Learning Group
Einsteinufer 37, 10587 Berlin, Germany
Mail: [email protected]
More information: http://iphome.hhi.de/samek