first part: deep learning for speech recognition · 2017. 4. 4. · a guy on a skate board on the...
TRANSCRIPT
First part: Deep Learning for Speech recognition
Deep Speech
Acoustic model:Hidden Markov Model
Decoding:Viterbi algorithm
Why DL now?
Sequence 2 Sequence
Compressing Neural Nets
ICLR 2016 Best Paper award
Deep Compression results
ImageNet Results
The First Deep Topologies
Deep-Compression algorithm
Pruningconnections with weights below a threshold are removed
Retrain
Convolution as Matrix multiplication
Weight Sharing & Quantization
Huffman Coding
In Summary
The main component is Pruning We can replace it with SVD:
Singular Value Decomposition
Splitting one layer into 3 layers
Original # parameters=m*nNew # parameters= m*r+r^2 +r*n
2015
Squeeze-net (2015)• Decompose the filters layers into smaller filters
• Instead of having 7x3x3 (147) parameters we have 3x1x1+4x1x1+4x3x3 (43)
Squeeze-net topology:late maxpooling
Squeeze-net topology:Deep Compression
Squueze-net Results
What about new topologies?
Compression barely works..
Why it isn’t working?The convolutional layers already use the “Squeeze-net “ trick
Does compression has a value?:Weight Sharing & Quantization still work...
Does compression has a value? Detection
Does compression has a value?Segmentation
Does compression has a value? Image Captioning
Why shallow networks outperform the deeper ones?
Promising Directions