introduction to deep learning (dmytro fishman technology stream)
TRANSCRIPT
Introduction to Deep LearningDmytro Fishman, University of Tartu
What is Machine Learning?Was born with a goal to create Artificial Intelligence !There were two main branches: • Symbolic AI, based on if/then like rules (GOFAI) !• Inspired by Neuroscience principle - learn through
examples (later developed into ML)
Machine Learning founding disciplines
Computer Science
Statistics
Information Theory Databases
Probability Theory
Optimisation
Machine Learning
What is Machine Learning?
Machine learning is a scientific field that develops computer programs that make data-driven decisions without being explicitly programmed and also automatically improve over time.
What is Machine Learning?
Machine learning is • a scientific field • develops computer programs • make data-driven decisions • without being explicitly programmed to • also automatically improve over time.
Examples of Machine LearningSpam Filtering
Examples of Machine LearningFace Recognition
Examples of Machine LearningRecommendation Systems
Comes from nature…
http://www.borgenmagazine.com/wp-content/uploads/2014/06/natural_disasters.jpg
…and industryhttp://forums.vwvortex.com/showthread.php?3208922-2008-Dodge-Challenger-Factory-production-pics./page2
Often is very messy
http://insights.wired.com/profiles/blogs/what-s-the-big-deal-with-unstructured-data
Ideally a long !list of examples of the same nature
Types of data
255,255,255,255 255,255,255,255 255, 56, 65,255 255, 98, 89,110
To be or not to be.
x = (0.5, 0.9, 0.7, -0.3, …)
x = (255, 255, 255, 255, …)
x = (0, …, 0, 2, 1, 0, 0, 1, 2, …)
to or not
bethe
cat
low John
Adopted from P.Vincent http://videolectures.net/deeplearning2015_vincent_machine_learning/
Machine Learning tasksSupervised Learning • Classification • Regression!!Unsupervised Learning!• Clustering • Dimensionality reduction • Density Estimation !Reinforcement Learning - taking actions in an environment to maximise reward
Machine Learning tasksSupervised Learning • Classification • Regression!!Unsupervised Learning!• Clustering • Dimensionality reduction • Density Estimation !Reinforcement Learning - taking actions in an environment to maximise reward
Supervised LearningCat!
Cat
???
Dog
x1 = (87, 205, 255, 155, …); y1 = 1
x2 = (95, 195, 245, 155, …); y2 = 1
x3 = (43, 159, 255, 5, …); y3 = -1
x4 = (180, 66, 245, …); !y4 = f(x4) = 1
Features Labels
Adopted from P.Vincent http://videolectures.net/deeplearning2015_vincent_machine_learning/
Evolution of ML methodsRule-based Classic Machine
LearningRepresentation
Learning Deep Learning
Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/
Input Input Input Input
Fixed set of rules
Output
Hand designed features
Learning
Automated feature extraction
Output
Learning
Output
Low level features
High level features
Learning
Output
Mimicking the brainEngineering perspective • Compact!• Energy efficient (12 watts—a fifth of
the power of a lightbulb) • 10^11 neurons • for each neurone there are 10-50
supporting cells (glial cells) • 10^14 synapses (connections)
Computing power • Fairly slow in arithmetic calculations • Very efficient in computer vision, speech, language etc.
Adopted from L.Bottou http://videolectures.net/deeplearning2015_bottou_neural_networks/
Biological Neuron
Caution! Here comes the slide with
some math!
Artificial Neuronex0
w0
x1w1
x2w2
X
i
xiwi + b
f
axon from neuronedendrite
synapse
output axonactivation!function
f(X
i
xiwi + b)
Slide from CS231n: Convolutional Neural Networks for Visual Recognition http://cs231n.github.io/ by Andrej Karpathy
Feed Forwardx0
w0
x1w1
x2w2
X
i
xiwi + b
f
axon from neuronedendrite
synapse
output axonactivation!function
f(X
i
xiwi + b)
Slide from CS231n: Convolutional Neural Networks for Visual Recognition http://cs231n.github.io/ by Andrej Karpathy
class Neuron: # ... def forward(inputs): """ assume inputs and weights are 1-D numpy arrays and bias is a number """ cell_body_sum = np.sum(inputs * self.weights) + self.bias firing_rate = 1.0 / (1.0 + math.exp(-cell_body_sum)) # sigmoid activation function return firing_rate
Training Neural Netowrks
43
255
32
Output
Hidden
…
Input
Image
32
143
255
Output
Hidden
…
Input
Image
Training Neural Netowrks
32
143
255
Output
Hidden
…
Input
Image
Training Neural Netowrks
0.6
32
143
Input
255
0.4
OutputHidden
Image
…Cat
Dog
Error
Training Neural Netowrks
Deep Neural Networks
GoogLeNet
Types of Deep Neural Networks
• Convolutional Neural Networks (CNN) • Convolutional Deep Belief Networks (CDBN)
!• Recurrent Neural Networks (RNN)
• Long Short-Term Memory Networks (LSTM) !
https://en.wikipedia.org/wiki/Deep_learning
Convolutional Neural Networks
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
Breed?
http://cs.stanford.edu/people/karpathy/ilsvrc/
Misclassified examples
Overall:trained expert got 5.1% of errorsGoogLeNet 6.7% of errorsBefore Deep Neural Networks the best result was 20.7%
Galaxy Zoo Challenge
61578 JPG images, made by Hubble
Galaxy Zoo Challenge
http://benanne.github.io/2014/04/05/galaxy-zoo.html
Diabetic Retinopathy Detection Challenge
0 stage
4 stage
and 80 000 more images
Diabetic Retinopathy Detection Challenge
…
…
Diabetic Retinopathy Detection Challenge
DeepMind
DeepMind
0.6
32
143
Input
255
0.4
OutputHidden
Image
…Right
Left
Lost one life?
Training DeepMind
Recurrent Neural Networks
Shakespeare?Recurrent Neural Network
PANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep. !Second Senator: They are away this miseries, produced upon my soul, Breaking and strongly should be buried, when I perish The earth and thoughts of many states. !DUKE VINCENTIO: Well, your wit is in the care of side and that. !Second Lord: They would be ruled after this chamber, and my fair nues begun out of the fact, to be conveyed, Whose noble souls I'll have the heart of the wars. !Clown: Come, sir, I will make did behold your worship. !VIOLA: I'll drink it.
VIOLA. On your attendance, my lord; here. !DUKE. Stand you awhile aloof.—Cesario, Thou know'st no less but all; I have unclasp'd To thee the book even of my secret soul: Therefore, good youth, address thy gait unto her; Be not denied access, stand at her doors, And tell them there thy fixed foot shall grow Till thou have audience. !VIOLA. Sure, my noble lord, If she be so abandon'd to her sorrow As it is spoke, she never will admit me. !DUKE. Be clamorous and leap all civil bounds, Rather than make unprofited return. !VIOLA. Say I do speak with her, my lord. What then?
4.4МB Shakespeare
texts
Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy
Recurrent Neural NetworkPANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep. !Second Senator: They are away this miseries, produced upon my soul, Breaking and strongly should be buried, when I perish The earth and thoughts of many states. !DUKE VINCENTIO: Well, your wit is in the care of side and that. !Second Lord: They would be ruled after this chamber, and my fair nues begun out of the fact, to be conveyed, Whose noble souls I'll have the heart of the wars. !Clown: Come, sir, I will make did behold your worship. !VIOLA: I'll drink it.
VIOLA. On your attendance, my lord; here. !DUKE. Stand you awhile aloof.—Cesario, Thou know'st no less but all; I have unclasp'd To thee the book even of my secret soul: Therefore, good youth, address thy gait unto her; Be not denied access, stand at her doors, And tell them there thy fixed foot shall grow Till thou have audience. !VIOLA. Sure, my noble lord, If she be so abandon'd to her sorrow As it is spoke, she never will admit me. !DUKE. Be clamorous and leap all civil bounds, Rather than make unprofited return. !VIOLA. Say I do speak with her, my lord. What then?
Shakespeare?
Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy
Linux 474MB of the source code
/*! * Increment the size file of the new incorrect UI_FILTER group information! * of the size generatively.! */!static int indicate_policy(void)!{! int error;! if (fd == MARN_EPT) {! /*! * The kernel blank will coeld it to userspace.! */! if (ss->segment < mem_total)! unblock_graph_and_set_blocked();! else! ret = 1;! goto bail;! }! segaddr = in_SB(in.addr);! selector = seg / 16;! setup_works = true;! for (i = 0; i < blocks; i++) {! seq = buf[i++];! bpf = bd->bd.next + i * search;! if (fd) {! current = blocked;! }! }! rw->name = "Getjbbregs";! bprm_self_clearl(&iv->version);! regs->new = blocks[(BPF_STATS << info->historidac)] | PFMR_CLOBATHINC_SECONDS << 12;! return segtable;!}
Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy
Tools• Python (many implementations) • Torch (http://torch.ch/) • Theano (http://deeplearning.net/software/theano/) • Caffe (http://caffe.berkeleyvision.org/)
!!
Deep Learning Libraries by programming language: http://www.teglor.com/b/deep-learning-libraries-language-cm569
Online courses
CS224d: Deep Learning for Natural Language Processing by Richard Socher http://cs224d.stanford.edu/
Machine Learning by Andrew Ng https://www.coursera.org/learn/machine-learning
Neural Networks for Machine Learning by Geoffrey Hinton https://www.coursera.org/course/neuralnets
CS231n: Convolutional Neural Networks for Visual Recognition by Andrej Karpathy http://cs231n.github.io/
Offline placesComputer Science Bachelor’s program at CS@UCU http://cs.ucu.edu.ua/en/
Applied Computer Science Stream at Summer School AACIMP (Kyiv) http://summerschool.ssa.org.ua/
The Deep Learning Summer School at Montreal, Canada https://sites.google.com/site/deeplearningsummerschool/
Computer Science Master’s program at Tartu, Estonia https://sites.google.com/site/deeplearningsummerschool/
International Conference on Machine Learning at New York, USA http://icml.cc/2016/
Popular Blogs10 Machine Learning Terms Explained in Simple English by Aylien http://blog.aylien.com/post/121281850733/10-machine-learning-terms-explained-in-simple
The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy http://karpathy.github.io/2015/05/21/rnn-effectiveness/
A Neural Network in 11 lines of Python by iamtrask http://iamtrask.github.io/2015/07/12/basic-python-network/
Understand LSTM Networks by Christopher Olah https://colah.github.io/posts/2015-08-Understanding-LSTMs/
Sign up for Data Elixir mailing list: http://dataelixir.com/
KagglePlatform for data science competitions
Sponsored
Research
Public competitions
Computers are useless. They can only give answers.
Pablo Picasso