Deep Learning

Roberto Paredes (rparedes@prhlt.upv.es)PRHLT Research Center

Universitat Politecnica de Valencia

March 2016

Deep Learning Introduction

• Neural networks

• Deep Learning: Stack many layers to build deep models

• Recently, grab the attention of the industry

• Many new applications

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 2

Deep Learning Introduction

http://qz.com/335768/bill-gates-joins-elon-musk-and-stephen-hawking-in-saying-artificial-intelligence-is-scary/

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 3

Deep Learning Introduction

• Key issue: Representational Learning

• Seamless Representation-Classification model

... and make it happen!

... and even make it affordable!

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 4

Deep Learning Introduction

• Neural Network:

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 5

Deep Learning Introduction

• Deep Learning → Bridge the gap between raw representation and categories

http://www.clarifai.com/static/img_ours/cnn.png

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 6

Deep Learning Applications

• Deep Learning:

– Key issue Representational Learning– Some realistic problems require a deep structure to be learned properly

• Applications:

– Image Recognition– Speech Recognition– Natural Language Processing– Machine Translation– ...

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 7

Deep Learning Achievements - Computer Vision

• ImageNet Challenge

http://blogs.nvidia.com/blog/2014/09/18/gpus-imagenet-deep-learning/

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 8

Deep Learning Achievements - Computer Vision and NLP

http://googleresearch.blogspot.com.es/2014/11/a-picture-is-worth-thousand-coherent.html

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 9

Deep Learning Achievements - Computer Vision and NLP

http://googleresearch.blogspot.com.es/2014/11/a-picture-is-worth-thousand-coherent.html

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 10

Deep Learning Achievements - Computer Vision

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional

neural networks. In Advances in neural information processing systems (pp. 1097-1105).

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 11

Deep Learning Achievements - Computer Vision

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 12

Deep Learning Achievements - Computer Vision

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 13

Deep Learning Achievements - Speech Recognition

• Tandem DBN-DNN-HMM

https://www.cs.toronto.edu/~hinton/absps/DNN-2012-proof.pdf

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 14

Deep Learning Achievements - Handwritten Text Recognition

• Bidirectional LSTM. CTC and LM

• ICDAR competition, WER: from 27 (Basic HMM) down to 15 (Combinations)

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 15

Deep Learning (Neural Networks) Approaches

• Neural Network (Multi-Layer Perceptron)

– Deep Neural Network (Multi-Layer Perceptron with more layers)– Word2Vec (NLP) (Multi-Layer Perceptron)– Autoencoders and Denoising Autoencoders (Multi-Layer Perceptron)

• Convolutional Neural Networks

• Long-Short Term Memory LSTM

• Restricted Boltzmann Machines

– Deep Belief Networks– Deep Boltzmann Machines

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 16

Deep Learning (Neural Networks) Approaches (Dates)

• Neural Network (Multi-Layer Perceptron) (1986)

• Convolutional Neural Networks (1989)

• Long-Short Term Memory LSTM (1997)

• Restricted Boltzmann Machines (2006)

– Deep Belief Networks– Deep Boltzmann Maniches

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 17

Deep Learning (Neural Networks) approaches

• Supervised:

– DNN– Convolutional NN– LSTM

Training: Backpropagation

• Unsupervised:

– Stacked Autoencoders

Training: Backpropagation

– Restricted Boltzmann Machines

Contrastive Divergence or Persistent CD

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 18

Deep Learning problems

• Problem when backpropagating errors to first layers

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 19

Deep Learning problems

• Nowadays a DeepNet for computer vision is something like this:

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 20

Deep Learning problems

• Backpropagation , Why now it works with deep structures?

– Tons of data– Sharing weights on very initial layers (CNN)– Improving Generalization:∗ Dropout∗ Dropconnect∗ Denoising Autoencoders

– New activation functions:∗ ReLU∗ MaxOut⇒ piecewise-linear behaviour and constant gradients

– Layer by layer training– Virtual Data with common distortions– Hardware allows to run experiments!! (GPU)

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 21

Deep Learning things to consider

• Supervised training (DNN,CNN):

– Network topology– Sigmoid, tanh, ReLu, softmax, linear, Maxout– Data normalization– Virtual data– Batch size, Epochs– Weights initialization– Learning rate– Momentum rate– Dropout– Dropconnect– ...

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 22

Deep Learning, when?

• When to use Deep Learning:

– There is an big gap between raw representation and categories– Hand-crafted features didn’t work– There are a lot of data for training (or virtual distortions)– Good hardware is available

• When hand-crafted features are good (expert knowledge):

– SVM– Random Forests, ERT– AdaBoost– ...

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 23

Deep Learning

Thanks for your attention

March 15, 2016 - PAPIs Connect, Valencia Deep Learning Page 24