neural networks, spark mllib, deep learning

87
NEURAL NETWORKS AND DEEP LEARNING BY ASIM JALIS GALVANIZE

Upload: asim-jalis

Post on 21-Apr-2017

6.674 views

Category:

Data & Analytics


16 download

TRANSCRIPT

Page 1: Neural Networks, Spark MLlib, Deep Learning

NEURAL NETWORKS AND DEEPLEARNING

BY ASIM JALISGALVANIZE

Page 2: Neural Networks, Spark MLlib, Deep Learning

WHO AM I?

Page 3: Neural Networks, Spark MLlib, Deep Learning

ASIM JALISGalvanize/Zipfian, Data EngineeringCloudera, Microso!, SalesforceMS in Computer Science from University of Virginiahttps://www.linkedin.com/in/asimjalis

Page 4: Neural Networks, Spark MLlib, Deep Learning

WHAT IS GALVANIZE’S DATAENGINEERING PROGRAM?

Page 5: Neural Networks, Spark MLlib, Deep Learning
Page 6: Neural Networks, Spark MLlib, Deep Learning

DO YOU WANT TO . . .Play with terabytes of dataBuild data applications using Spark, Hadoop, Hive, Kafka,Storm, HBaseUse Data Science algorithms at scale

Page 7: Neural Networks, Spark MLlib, Deep Learning

WHAT IS INVOLVED?Learn concepts in interactive lecturesDevelop skills in hands-on labsDesign and build your Capstone ProjectShow project to SF tech companies at Hiring Day

Page 8: Neural Networks, Spark MLlib, Deep Learning

FOR MORE INFORMATIONCheck out Talk to me

http://galvanize.com

[email protected]

Page 9: Neural Networks, Spark MLlib, Deep Learning

INTRO

Page 10: Neural Networks, Spark MLlib, Deep Learning

WHAT IS THIS TALK ABOUT?What are Neural Networks and how do they work?What is Deep Learning?What is the difference?How can we build neural networks in Apache Spark?

Page 11: Neural Networks, Spark MLlib, Deep Learning

HOW MANY PEOPLE HERE AREFAMILIAR WITH NEURAL

NETWORKS?

Page 12: Neural Networks, Spark MLlib, Deep Learning

HOW MANY PEOPLE HERE AREFAMILIAR WITH CONVOLUTION

NEURAL NETWORKS?

Page 13: Neural Networks, Spark MLlib, Deep Learning

HOW MANY PEOPLE HERE AREFAMILIAR WITH DEEP LEARNING?

Page 14: Neural Networks, Spark MLlib, Deep Learning

HOW MANY PEOPLE HERE AREFAMILIAR WITH APACHE SPARK

AND MLLIB?

Page 15: Neural Networks, Spark MLlib, Deep Learning

NEURAL NETWORKS

Page 16: Neural Networks, Spark MLlib, Deep Learning

WHAT IS A NEURON?

Page 17: Neural Networks, Spark MLlib, Deep Learning

Receives signal on synapseWhen trigger sends signal on axon

Page 18: Neural Networks, Spark MLlib, Deep Learning

Mathematical abstractionInspired by biological neuronEither on or off based on sum of input

Page 19: Neural Networks, Spark MLlib, Deep Learning

Neuron is a mathematical functionAdds up (weighted) inputsApplies the sigmoid functionThis determines if it fires or not

Page 20: Neural Networks, Spark MLlib, Deep Learning

WHAT ARE NEURAL NETWORKS?Biologically inspired machine learning algorithmMathematical neurons arranged in layersAccumulate signals from the previous layerFire when signal reaches threshold

Page 21: Neural Networks, Spark MLlib, Deep Learning
Page 22: Neural Networks, Spark MLlib, Deep Learning

HOW MANY NEURONS SHOULD IHAVE IN MY NETWORK?

Page 23: Neural Networks, Spark MLlib, Deep Learning

HOW MANY INPUT LAYERNEURONS SHOULD WE HAVE?

Page 24: Neural Networks, Spark MLlib, Deep Learning

The number of inputs or features

Page 25: Neural Networks, Spark MLlib, Deep Learning

HOW MANY OUTPUT LAYERNEURONS SHOULD WE HAVE?

Page 26: Neural Networks, Spark MLlib, Deep Learning

The number of classes we are classifying the input into.

Page 27: Neural Networks, Spark MLlib, Deep Learning

HOW MANY HIDDEN LAYERNEURONS SHOULD WE HAVE?

Page 28: Neural Networks, Spark MLlib, Deep Learning

SIMPLEST OPTION IS TO USE 0.

Page 29: Neural Networks, Spark MLlib, Deep Learning

SINGLE LAYER PERCEPTRON

Page 30: Neural Networks, Spark MLlib, Deep Learning

WHAT ARE THE DOWNSIDES OFNO HIDDEN LAYERS?Only works if data is linearly separable.Identical to logistic regression.

Page 31: Neural Networks, Spark MLlib, Deep Learning

MULTILAYER PERCEPTRONFor most realistic classification tasks you will need ahidden layer.Rule of thumb:

Number of hidden layers equals oneNumber of neurons in hidden layer is mean of size ofinput and output layers.

Page 32: Neural Networks, Spark MLlib, Deep Learning

HOW DO WE USE THIS THING?

Page 33: Neural Networks, Spark MLlib, Deep Learning

NEURAL NETWORK WORKFLOWSplit labeled data into train and test setsTrain with labeled dataTest and compare prediction with actual labels

Page 34: Neural Networks, Spark MLlib, Deep Learning

HOW DO WE TRAIN IT?

Page 35: Neural Networks, Spark MLlib, Deep Learning

FEED FORWARDAlso called forward propagation or forward propInitialize inputsWeigh inputs into hidden layer, sum, apply sigmoidCalculate activation of hidden layerWeight inputs into output layer, sum, apply sigmoidCalculate activation of output layer

Page 36: Neural Networks, Spark MLlib, Deep Learning

BACK PROPAGATIONUse forward prop to calculate the errorError is function of all network weightsAdjust weights using gradient descentRepeat with next recordKeep going over training set until convergence

Page 37: Neural Networks, Spark MLlib, Deep Learning

WHAT IS GRADIENT DESCENT?

Page 38: Neural Networks, Spark MLlib, Deep Learning
Page 39: Neural Networks, Spark MLlib, Deep Learning

HOW DO YOU FIND THE MINIMUMIN AN N-DIMENSIONAL SPACE?

Take a step in the steepest direction.Steepest direction is vector sum of all derivatives.

Page 40: Neural Networks, Spark MLlib, Deep Learning

PUTTING ALL THIS TOGETHER

Page 41: Neural Networks, Spark MLlib, Deep Learning

Use forward prop to activateUse back prop to trainThen use forward prop to test

Page 42: Neural Networks, Spark MLlib, Deep Learning

WHY NOT HAVE MULTIPLELAYERS?

Page 43: Neural Networks, Spark MLlib, Deep Learning

DOWNSIDE OF MULTIPLE LAYERSNumber of weights is a product of the layer sizesThe mathematics quickly becomes intractableParticularly when your input is an image with tens ofthousands of pixels

Page 44: Neural Networks, Spark MLlib, Deep Learning

APACHE SPARK MLLIB

Page 45: Neural Networks, Spark MLlib, Deep Learning

WHAT IS SPARK

Page 46: Neural Networks, Spark MLlib, Deep Learning
Page 47: Neural Networks, Spark MLlib, Deep Learning

Framework for processing data across a clusterBy sending the code to the dataAnd executing the code where the data lives

Page 48: Neural Networks, Spark MLlib, Deep Learning

WHAT IS MLLIB?Library for Machine Learning.Builds on top of Spark RDDs.Provides RDDs for Machine Learning.Implements common Machine Learning algorithms.

Page 49: Neural Networks, Spark MLlib, Deep Learning
Page 50: Neural Networks, Spark MLlib, Deep Learning

DEMO USING APACHE TOREE

Page 51: Neural Networks, Spark MLlib, Deep Learning

WHAT IS APACHE TOREE?Like IPython Notebook but for Spark/Scala.Jupyter kernel for Spark/Scala.

Page 52: Neural Networks, Spark MLlib, Deep Learning

HOW CAN I INSTALL TOREE?Use pip to install IPython or Jupyter.Install Apache Spark by downloading tgz file andexpanding.SPARK_HOME=$HOME/spark-1.6.0pip install toreejupyter toree install \--spark_home=$SPARK_HOME

Page 53: Neural Networks, Spark MLlib, Deep Learning

HOW CAN I RUN A TOREENOTEBOOK

jupyter notebookVisit Create new notebook.Set kernel to Toree.sc in notebook should print Spark Context.

http://localhost:8888

Page 54: Neural Networks, Spark MLlib, Deep Learning

NEURAL NETWORKCONSTRUCTION

Page 55: Neural Networks, Spark MLlib, Deep Learning

HOW CAN I FIGURE OUT HOWMANY LAYERS?

To figure out how many layers to use and what topologyto use you have to rely on standard machine learningtechniques.Use cross-validation.In general k-fold cross validation.10-fold cross validation is popular.

Page 56: Neural Networks, Spark MLlib, Deep Learning

WHAT IS 10-FOLD CROSSVALIDATION OR K-FOLD CROSS

VALIDATION?

Page 57: Neural Networks, Spark MLlib, Deep Learning

Split your data into 10 (or in general k) equal-sizedsubsets.Train model on 9 of them, set one aside for cross-validation.Validate model on 10th and remember your error rate.Repeat by setting aside each one of the 10.Average the 10 error rates.Then repeat for the next model.Choose the model with the lowest error rate.

Page 58: Neural Networks, Spark MLlib, Deep Learning

HOW DO I DEPLOY MY NEURALNETWORK INTO PRODUCTION?There are two phases.The training phase can be run on the back-end servers.Cross-validate your model and its hyper-parameters onthe back-end.Then deploy the model to the front-end servers, browsers,devices.The front-end only uses forward prop and is always fast.

Page 59: Neural Networks, Spark MLlib, Deep Learning

DEEP LEARNING

Page 60: Neural Networks, Spark MLlib, Deep Learning

WHAT IS DEEP LEARNING?Deep Learning is a learning method that can train thesystem with more than 2 or 3 non-linear hidden layers.

Page 61: Neural Networks, Spark MLlib, Deep Learning

WHAT IS DEEP LEARNING?Machine learning techniques which enable unsupervisedfeature learning and pattern analysis/classification.The essence of deep learning is to computerepresentations of the data.Higher-level features are defined from lower-level ones.

Page 62: Neural Networks, Spark MLlib, Deep Learning

HOW IS DEEP LEARNINGDIFFERENT FROM REGULAR

NEURAL NETWORKS?Training neural networks requires applying gradientdescent on millions of dimensions.This is intractable for large networks.Deep learning places constraints on neural networks.This allows them to be solvable iteratively.The constraints are generic.

Page 63: Neural Networks, Spark MLlib, Deep Learning

WHAT IS THE BIG DEAL ABOUTIT?

AlexNet submitted to the ImageNet ILSVRC challenge in2012 is partly responsible for the renaissance.Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton usedDeep Learning techniques.They combined this with GPUs, some other techniques.The result was a neural network that could classify imagesof cats and dogs.It had an error 16% compared to 26% for the runner up.

Page 64: Neural Networks, Spark MLlib, Deep Learning

ILYA SUTSKEVER, ALEXKRIZHEVSKY, GEOFFREY HINTON

Page 65: Neural Networks, Spark MLlib, Deep Learning
Page 66: Neural Networks, Spark MLlib, Deep Learning

WHAT ARE THE DIFFERENT KINDSOF DEEP ARCHITECTURES?

GenerativeDiscriminativeHybrid

Page 67: Neural Networks, Spark MLlib, Deep Learning

WHAT ARE GENERATIVEARCHITECTURES

Extract features from dataFind common features in unlabelled dataLike Principal Component AnalysisUnsupervised: no labels required

Page 68: Neural Networks, Spark MLlib, Deep Learning

WHAT ARE DISCRIMINATIVEARCHITECTURES

Classify inputs into classesRequire labelsRequire supervised training

Page 69: Neural Networks, Spark MLlib, Deep Learning

WHAT ARE HYBRIDARCHITECTURES?

STEP 1Combination of generative and discriminativeExtract features using generative networkUse unsupervised learning

STEP 2Train discriminative network on extracted featuresUse supervised learning

Page 70: Neural Networks, Spark MLlib, Deep Learning

WHAT ARE AUTO-ENCODERS?An auto-encoder is a learning algorithm.It applies backpropagation and sets the target values tobe equal to its inputs.In other words it trains itself to do the identitytransformation.

Page 71: Neural Networks, Spark MLlib, Deep Learning
Page 72: Neural Networks, Spark MLlib, Deep Learning

WHY DOES IT DO THIS?By placing constraints on it, like restricting the number ofhidden neurons, it can find a good representation of thedata.

Page 73: Neural Networks, Spark MLlib, Deep Learning

IS THE AUTO-ENCODERSUPERVISED OR UNSUPERVISED?

Page 74: Neural Networks, Spark MLlib, Deep Learning

It is unsupervised.The data is unlabeled.Auto-encoders are similar to PCA (Principal ComponentAnalysis).PCA is a technique for reducing the dimensions of data.

Page 75: Neural Networks, Spark MLlib, Deep Learning

WHAT ARE CONVOLUTIONNEURAL NETWORKS?

Feedforward neural networks.Connection pattern inspired by visual cortex.

Page 76: Neural Networks, Spark MLlib, Deep Learning
Page 77: Neural Networks, Spark MLlib, Deep Learning

CONVOLUTION NEURALNETWORKS

The convolution layer’s parameters are a set of learnablefilters.Every filter is small along width and height.During the forward pass, each filter slides across the widthand height of the input, producing a 2-dimensionalactivation map.As we slide across the input we compute the dot productbetween the filter and the input.

Page 78: Neural Networks, Spark MLlib, Deep Learning

CONVOLUTION NEURALNETWORKS

Intuitively, the network learns filters that activate whenthey see a specific type of feature anywhere.In this way it creates translation invariance.

Page 79: Neural Networks, Spark MLlib, Deep Learning

WHAT IS A POOLING LAYER?The pooling layer reduces the resolution of the imagefurther.It tiles the output area with 2x2 mask and takes themaximum activation value of the area.

Page 80: Neural Networks, Spark MLlib, Deep Learning
Page 81: Neural Networks, Spark MLlib, Deep Learning

DOES SPARK SUPPORT DEEPLEARNING?

Not directly yethttps://issues.apache.org/jira/browse/SPARK-2352

Page 82: Neural Networks, Spark MLlib, Deep Learning

WHAT ARE SOME MAJOR DEEPLEARNING PLATFORMS?

Page 83: Neural Networks, Spark MLlib, Deep Learning

Theano: Low-level GPU-enabled tensor library.Lasagne, Blocks: NN libraries that make Theano easier touse.Torch7: NN library. Uses Lua for binding. Used byFacebook and Google.Caffe: NN library by Berkeley AMPLab.Pylearn2: ML library based on Theano by University ofToronto. Google DeepMind.cuDNN: NN library by Nvidia based on CUDA. Can be usedwith Torch7, Caffe.Chainer: NN library that uses CUDA.TensorFlow: NN library from Google.

Page 84: Neural Networks, Spark MLlib, Deep Learning

WHAT LANGUAGE ARE THESE IN?All the frameworks support Python.Except Torch7 which uses Lua for its binding language.

Page 85: Neural Networks, Spark MLlib, Deep Learning

WHAT CAN I DO ON SPARK?SparkNet: Integrates running Caffe with Spark.Sparkling Water: Integrates H2O with Spark.DeepLearning4J: Built on top of Spark.TensorFlow on Spark (experimental)

Page 86: Neural Networks, Spark MLlib, Deep Learning

QUESTIONS

Page 87: Neural Networks, Spark MLlib, Deep Learning

GALVANIZE DATA ENGINEERING