methods and resources distributed deep learningsergey/slides/n17_aiukrainedistributed.pdf · 10...

Distributed Deep Learning:Methods and Resources

Sergey NikolenkoChief Research Officer, Neuromation

Researcher, Steklov Institute of Mathematics at St. Petersburg

September 23, 2017, AI Ukraine, Kharkiv

Maxim PrasolovCEO, Neuromation

Outline

● Bird’s eye overview of deep learning

● SGD and how to parallelize it

● Data parallelism and model parallelism

● Neuromation: developing a worldwide marketplacefor knowledge mining

● 10 years ago machine learning underwent a deep learning revolution

● Neural networks are one of the oldest techniques in ML

● But since 2007-2008, we can train large and deep neural networks(in part due to distributed computations)

● And now deep NNs yield state of the art results in many fields

What is a deep neural network

● A deep neural network is a huge composition of simple functionsimplemented by artificial neurons

● Usually linear combination followed by nonlinearity, but can beanything as long as you can takederivatives

● These functions are combined into acomputational graph that computes the loss function for the model

Backpropagation● To train the model (learn the weights),

you take the gradient of the lossfunction w.r.t. weights

● Gradients can be efficiently computedwith backpropagation

● And then you can do (stochastic)gradient descent and all of itswonderful modifications, fromNesterov momentum to Adam

FEEDFORWARD NETWORKS

CONVOLUTIONAL NETWORKS

RECURRENT NETWORKS

Gradient descent is used for all kinds of neural networks

Distributed Deep Learning: The Problem

● One component of the DL revolutionwas the use of GPUs

● GPUs are highly parallel (hundreds of cores)and optimized for matrix computations

● Which is perfect for backprop (and fprop too)

● But what if your model does not fit on a GPU?

● Or what if you have multiple GPUs?

● Can we parallelize further?

What Can Be Parallel

● Model parallelism vs. data parallelism

● We will discuss both

● Data parallelism is much more common

● And you can unite the two:

[pictures from (Black, Kokorin, 2016)]

Examples of data parallelism

● Make every worker do its thing andthen average the results

● Parameter averaging: average w from all workers○ but how often?○ and what do we do with advanced SGD variants?

● Asynchronous SGD: average updates from workers○ much more interesting without synchronization○ but the stale gradient problem

● Hogwild (2011): very simple asynchronous SGD,just read and write to shared memory, lock-free;whatever happens, happens

Examples of data parallelism

● FireCaffe:○ DP on a GPU cluster○ communication through

reduction trees

Model parallelism

● In model parallelism, different weights are distributed

● Pictures from the DistBelief paper (Dean et al., 2012)

● Difference in communication:○ DP: workers exchange weight updates ○ MP: workers exchange data updates

● DP in DistBelief: Downpour SGDvs. Sandblaster L-BFGS

● Now, DistBelief has been completely replaced by...

Distributed Learning in TensorFlow

● TensorFlow has both DP (right) and MP (bottom)

● Workers and parameter servers

● MP usually works as a pipeline between layers:

First specify the structure of the cluster: Then assign (parts of) computational graph to workers and weights to parameter servers:

Example of Data Parallelism in TensorFlow

Interesting variations

● (Zhang et al., 2016) – staleness-aware SGD: add weights depending on the time (staleness) to updates

● Elephas: distributed Keras that runs on Spark

● (Xie et al., 2015) – sufficient factor broadcasting:represent and send only u and v

● (Zhang et al., 2017) – Poseidon: a new architecture withwait-free backprop and hybrid communication

● Special mention: reinforcement learning; async RL is great!● And standard (by now) DQN tricks are perfect for parallelization:

○ experience replay: store experience in replay memory and serve them for learning

○ target Q-network is separate from the Q-network which is learning now, updates are rare

Distributed reinforcement learning

Gorila from DeepMind: everything is parallel and asynchronous

Recap

● Data parallelism lets you process lots of data in parallel, copying the model

● Model parallelism lets you break down a large model into parts

● Distributed architectures are usually based on parameter servers and workers

● Especially in reinforcement learning, where distributed architectures rule

● And this all works out of the box in TensorFlow and other modern frameworks

● But how is it relevant to us? Isn’t that for the likes of Google and/or DeepMind?

● Where do we get the computational power and why do we need so much data?

Distributed deep learning works

BITCOIN OR ETHER MINING

AMAZON DEEP LEARNING

$7-8 USDper DAY

$3-4 USDHOUR

Chris

NOT ENOUGH LABELED DATA FOR NEURAL NETWORK TRAINING

“BOTTLENECK” OF AUTOMATION OF EVERY INDUSTRY:



$7-8 USDper DAY

$3-4 USDHOUR

IMAGE RECOGNITION IN RETAIL



$7-8 USDper DAY

$3-4 USDHOUR

170.000 OBJECTS

ABOUT 40 BLN IMAGES PER YEAR

TO AUTOMATE THE RETAIL INDUSTRY

MUST BE RECOGNIZED ON THE SHELVES

IMAGE RECOGNITION IN RETAIL BY ECR RESEARCH:

OSA HP CONTRACTED NEUROMATION TO PRODUCE LABELED DATA AND TO RECOGNIZE

30% OF THE COST IS COMPUTATIONAL POWER.

MORE THAN 7 MLN EURO REVENUE



$7-8 USDper DAY

$3-4 USDHOUR

ChrisLABELED PHOTOS ARE REQUIRED TO TRAIN IMAGE RECOGNITION MODELS

MORE THAN

1 BLN

WHERE CAN WE GET THIS HUGE AMOUNT OF LABELED DATA?

Chris

1 MAN = 8 HOURS x 50 IMAGES, x $0.2 PER IMAGE

YEARS OF MECHANICAL WORK

1 BLN LABELED PHOTOS = $240 MLN

DATA LABELING HAS BEEN MANUAL WORK TILL NOW

WE KNOW HOW TO GENERATE SYNTHETIC LABELED DATA FOR DEEP LEARNING

● Labeled data with 100% accuracy

● Automated data generation with no limits

● Cheaper and faster than manual labor

SYNTHETIC DATA: A BREAKTHROUGH INDEEP LEARNING

BUT REQUIRES HUGE COMPUTATIONAL POWERTO RENDER DATA AND TRAIN NEURAL NETWORKS



$7-8 USDper DAY

$3-4 USDHOUR



$7-8per DAY

$3-4per HOUR

The AI industry is ready to pay miners for their computational resources more than they can ever get from mining Ether.

KNOWLEDGE MINING IS MORE PROFITABLE. DEEP LEARNING NEEDS YOUR COMPUTATION POWER!

WE CAN BRIDGE THIS GAP

GPU

x6GPU



$7-8 USDper DAY

$3-4 USDHOUR

BLOCKCHAIN + DEEP LEARNING



$7-8 USDper DAY

$3-4 USDHOUR

NEUROMATION PLATFORM

TokenAI

will combine in one place all the components necessary to build deep learning solutions with synthetic data

THE UNIVERSAL MARKETPLACE OF NEURAL NETWORK DEVELOPMENT



$7-8 USDper DAY

$3-4 USDHOUR

● the network must be geographically distributed and keep track of massive amounts of transactions

● the payment method for completed work should be highly liquid and politically independent

● network nodes have to understand the model of “mining” a resource for a bounty: transparency is required to build trust

● transactions need to be transparently auditable to prevent fraud and mitigate dispute

NEUROMATION PLATFORM will be extending Etherium with TokenAI.

Blockchain is the only technology that can realistically accomplish this.Extending Ethereum instead of building our own blockchain is an obvious first step.

Neuromation needs to quickly deploya massive network of computation nodes (converted from crypto miners).



$7-8 USDper DAY

$3-4 USDHOUR

DEEP LEARNING RESEARCH GRANTS

● for R&D teams and start-up’s of DL/ML industry

● In cooperation with frontier institutions

WE ARE OPEN FOR [email protected]

● 1000 GPUs pool (+100,000GPU are coming)

VAST APPLICATIONS OF SYNTHETIC DATA

NEUROMATION LABS:

RETAIL AUTOMATION LAB

PHARMA AND BIOTECH LAB

synthetic data for:

ENTERPRISE AUTOMATION LAB

synthetic data for:

● +170 000+ items for the Eastern European Retail Market only

● about 50 euro per object● contract for >7mln euro in

revenue

● medical imaging (classify tumors and melanomas)

● health applications (smart cameras)

● training flying drones, self-driving cars, and industrial robots in virtual environments

● manufacturing and supply-chain solutions

(live)

OUR TEAM:

Maxim PrasolovCEO

Fedor SavchenkoCTO

Sergey NikolenkoChief Research Officer

Denis PopovVP of Engineering

Constantine GoltsevInvestor / Chairman

Andrew RabinovichAdviser

Yuri KundinICO Compliance

Adviser

Aleksey SpizhevoiResearcher

Esther Katz VP Communication

Kiryl TruskovskyiLead Researcher

OCTOBER 15th, 2017 Presale of TOKENAI starts

NOVEMBER, 2017 Public sale starts

UNKNOWN DATE Secret cap is reached, and token sale ends in 7 days

Jan 1st, 2018 Token sale ends (if secret cap is not reached)

ICO ROADMAP

● Tokens Minted: 1,000,000,000 ● Issued in ICO: Up to 700,000,000

● Reserve: from 300,000,000

● Price per Token: 0.001 ETH

ICO DETAILS

Jurisdiction: Estonia, EU * Neuromation is fully compliant with Estonian crowdfunding legislation

KNOWLEDGE MINING - A NEW ERA OF DISTRIBUTED COMPUTING

THANK YOU!

neuromation.io

methods and resources distributed deep learningsergey/slides/n17_aiukrainedistributed.pdf · 10...

Documents