20151216 computer vision meetup deep learning deep learning - rene...deep learning . rené donner...
TRANSCRIPT
René Donner Deep Learning
Overview
2
The (amazing) things Deep Learning can do!
How does it work?
!
How can you start with DL?!
!
!
!
René Donner Deep Learning
Roughly …
3
Deep learning finds patterns in data corresponding to high-level, abstract concepts
!
!
!… just like we do, be we are better …
!
!
!… currently. In most tasks, not all, any more.
!
!
René Donner Deep Learning
What it can be used for
4
Image recognition
Text understanding, translation
Voice recognition
Playing video games
Driving cars
…
!
!
René Donner Deep Learning
Image recognition
5
Google Research Blog
René Donner Deep Learning
Image recognition
6
Google Image Search
René Donner Deep Learning
Scene labeling
7
http://www.purdue.edu/newsroom/releases/2014/Q1/smartphone-to-become-smarter-with-deep-learning-innovation.html
René Donner Deep Learning
Text recognition
8
!
http://www.pyimagesearch.com/2014/09/22/!getting-started-deep-learning-python/
Large-Scale Deep Learning for Intelligent Computer Systems, !Jeff Dean, Google, BayLearn 2015
René Donner Deep Learning
Text understanding
9
2013 Glove: Global Vectors for Word Representation, Jeffrey Pennington, Richard Socher and Christopher D. Manning
René Donner Deep Learning
Word embeddings
10
René Donner Deep Learning
Word embeddings
11
René Donner Deep Learning
Information extraction / Reasoning
12
MetaMind
René Donner Deep Learning
Information extraction / Reasoning
13
MetaMind
René Donner Deep Learning
It is not perfect
14
!
!
!
!
http://www.news.cornell.edu/stories/2015/03/images-fool-computer-vision-raise-security-concernsderStandard.at
René Donner Deep Learning
Some well know research groups
15
Stanford / BaiduAndrew Ng
!
NYU / FacebookYann LeCun
!
UToronto / GoogleGeoffrey Hinton
René Donner Deep Learning
NVIDIA
16
brand new: M40(same as Geforce GTX Titan X)
Images: NVIDIA website
How does it work?
René Donner Deep Learning
Difference to classic ML
18http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html
René Donner Deep Learning
Deep learning
19
http://theanalyticsstore.ie/deep-learning/
René Donner Deep Learning
Visualization
20
1. Layer
higher Layers
Emergence of Object-Selective Features in Unsupervised Feature Learning, Adam Coates, NIPS 2012
René Donner Deep Learning
Deep learning
21
How does it work?
http://theanalyticsstore.ie/deep-learning/ http://stats.stackexchange.com/questions/114385/!what-is-the-difference-between-convolutional-neural-networks-restricted-boltzma
René Donner Deep Learning
Optimization
22
Stochastic gradient descent
!
!
!
!
!
Automatic differentiation
blog.datumbox.com
René Donner Deep Learning
Local minima
23
Less problematic than thought - saddle points
https://ganguli-gang.stanford.edu/figures/14.Saddlepoint.jpg
René Donner Deep Learning
Deep learning
24
Low level features of color images
https://www.coursera.org/course/neuralnets
René Donner Deep Learning
Deep learning
25http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf
René Donner Deep Learning
ImageNet topologies
26
ImageNet Classification with Deep Convolutional Neural Networks", Alex Krizhevsky
“Inception” deep neural network architecture. Source: Christian Szegedy et. al. Going deeper with convolutions. CVPR 2015
René Donner Deep Learning
MNIST - Demo
27
René Donner Deep Learning
MNIST
28
http://deeplearning4j.org/rbm-mnist-tutorial.html
René Donner Deep Learning
Deep learning - why does it work?
29
Can cope with huge amounts of data
Learns small invariances
Overcomplete, sparse, representations
Learn Embedding
Lots of data
Recent advance: it is actually computable!
René Donner Deep Learning
Deep learning - pros
30
Not-domain specific
Supervised / Semi-supervised / Unsupervised
Classification / regression in last layer
Simple math
Hip
René Donner Deep Learning
Deep learning - cons
31
Lots of meta-parameters
Needs a lot of data
Very compute intensive
Hip
Getting started with DL
René Donner Deep Learning
Frameworks
33
Many different DL toolboxes
Efficiency important (GPU)
Attention to numerical issues
René Donner Deep Learning
Frameworks
34
Caffehttp://caffe.berkeleyvision.org/Plain text filesFastest CNN, GPU
!Keras
https://github.com/fchollet/kerasPython, on top of Theano
!TensorFlow
http://tensorflow.org/Python, by Google
!MXNet
https://github.com/dmlc/mxnetPython, R, Julia
Slid
e fro
m c
affe
tuto
rial
René Donner Deep Learning
Tensorflow
35
General gradient descent library
René Donner Deep Learning
Tutorials
36
Stanford tutorial:https://deeplearning.stanford.edu/wiki/index.php/UFLDL_TutorialMatlab code snippets
videolectures.nethttp://videolectures.net/deeplearning2015_montreal/
courserahttps://www.coursera.org/course/neuralnets
!
!
!
!
!
René Donner Deep Learning
Practical hints
37
Bengio ArxivPractical Recommendations for Gradient-Based Training of Deep Architectures, http://arxiv.org/abs/1206.5533http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html!
Kaggle http://www.kaggle.com/c/galaxy-zoo-the-galaxy-challengehttp://benanne.github.io/2014/04/05/galaxy-zoo.html
!
Relevant conferences NIPS (https://sites.google.com/site/deeplearningworkshopnips2013/accepted-papers)CVPR, ICMLMany interesting papers on arxiv.org
René Donner Deep Learning
Current research topics
38
Parallelization
What is deep learning, actually?
Alternative, faster, simpler methods
Multi-domain, transfer learning