Upload File

experiments with distributed training of neural networks on the grid

  • Home
  • Documents
  • Experiments with Distributed Training of Neural Networks on the Grid
1
Maciej Malawski 1 Marian Bubak 1,2 Elżbieta Richter-Wąs 3,4 Grzegorz Sala 3,5 Tadeusz Szymocha 3 1 Institute of Computer Science AGH, Mickiewicza 30, 30-059 Kraków, Poland 2 Academic Computer Centre CYFRONET, Nawojki 11, 30-950 Kraków, Poland 3 Institute of Nuclear Physics, Polish Academy of Sciences, Krakow, Poland 4 Institute of Physics, Jagiellonian University, Kraków, Poland 5 Faculty of Physics and Applied Computer Science AGH, Kraków, Poland {bubak,malawski}@agh.edu.pl, [email protected], [email protected], [email protected] Testbed for our experiments: EGEE project • Virtual Organization for Central Europe • CYFRONET Kraków, PSNC Poznań, KFKI Budapest, CESNET Prague, TU Kosice Grid sites • Support for MPI applications Why neural networks • Once trained, are efficient and accurate • Applicable for classification and prediction • Proven in wide area of applications Challenges • Neural network training is a highly compute- intensive task – may need High Performance Computing • Finding optimal configuration may be time consuming: many experiments with various parameters – may need High Throughput Computing Solution: The Grid • The distribution of the computation on a cluster of machines can lead to significant improvement in decreasing computation time. • Utilizing resources (multiple clusters) available on the Grid can make this task less time consuming for researcher. Target application • High Energy Physics • Discrimination between signal and background events coming from the particle detector (simulation) • ROOT and Athena as basic data analysis tools Observation Training of neural networks on the Grid requires many repeated tasks: • job preparation, • submission, • monitoring of status, • gathering results. Performing them manually is time consuming for the researcher → Preparation of tools for automating such tasks can facilitate the whole process considerably. Our Goals • Develop the tools facilitating usage of Grid for multiple classification experiments • Investigate and validate algorithms for distributed neural network training • Allow seamless integration with data analysis tools such as ROOT Nodei Update Weights Calculateerror Node 2 Update Weights Calculateerror Node 1 Update Weights Calculateerror Readthetrainingdata & thenetworkdetails Distributedata tothenodes Master repeat until stoppingcriterian ismet Backpropagationalgoritm Nodei Update Weights Calculateerror Nodei Update Weights Calculateerror Node 2 Update Weights Calculateerror Node 2 Update Weights Calculateerror Node 1 Update Weights Calculateerror Node 1 Update Weights Calculateerror Readthetrainingdata & thenetworkdetails Distributedata tothenodes Master repeat until stoppingcriterian ismet Backpropagationalgoritm Jacobian Hessjan Node i Jacobian Hessjan Node 2 Jacobian Hessjan Node 1 Readthetrainingdata & thenetworkdetails Distributedata tothenodes UpdateWeights Master repeat until stopping criterian ismet Levenberg-Marquardtalgoritm Jacobian Hessjan Node i Jacobian Hessjan Node 2 Jacobian Hessjan Node 1 Readthetrainingdata & thenetworkdetails Distributedata tothenodes UpdateWeights Master repeat until stopping criterian ismet Jacobian Hessjan Node i Jacobian Hessjan Node i Jacobian Hessjan Node 2 Jacobian Hessjan Node 2 Jacobian Hessjan Node 1 Jacobian Hessjan Node 1 Readthetrainingdata & thenetworkdetails Distributedata tothenodes UpdateWeights Master repeat until stopping criterian ismet Levenberg-Marquardtalgoritm Grid U ser Subm it() ClusterC ClusterB ClusterA Grid U ser Subm it() ClusterC ClusterC ClusterB ClusterB ClusterA ClusterA

Upload: rose-ellison

Post on 31-Dec-2015

29 views

Category:

Documents


2 download

Report

Tags:

DESCRIPTION

Experiments with Distributed Training of Neural Networks on the Grid. Maciej Malawski 1 Marian Bubak 1,2 Elżbieta Richter-Wąs 3,4 Grzegorz Sala 3,5 Tadeusz Szymocha 3 1 Institute of Computer Science AGH, Mickiewicza 30, 30-059 Kraków, Poland - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Experiments with Distributed Training of Neural Networks on the Grid

Maciej Malawski1 Marian Bubak1,2 Elżbieta Richter-Wąs3,4 Grzegorz Sala3,5 Tadeusz Szymocha3

1Institute of Computer Science AGH, Mickiewicza 30, 30-059 Kraków, Poland2Academic Computer Centre CYFRONET, Nawojki 11, 30-950 Kraków, Poland

3Institute of Nuclear Physics, Polish Academy of Sciences, Krakow, Poland4Institute of Physics, Jagiellonian University, Kraków, Poland

5Faculty of Physics and Applied Computer Science AGH, Kraków, Poland {bubak,malawski}@agh.edu.pl, [email protected], [email protected], [email protected]

Testbed for our experiments: EGEE project• Virtual Organization for Central Europe• CYFRONET Kraków, PSNC Poznań, KFKI

Budapest, CESNET Prague, TU Kosice Grid sites• Support for MPI applications

Why neural networks• Once trained, are efficient and

accurate• Applicable for classification and

prediction • Proven in wide area of applications

Challenges • Neural network training is a highly compute-

intensive task – may need High Performance Computing

• Finding optimal configuration may be time consuming: many experiments with various parameters – may need High Throughput Computing

Solution: The Grid• The distribution of the computation on a cluster of

machines can lead to significant improvement in decreasing computation time.

• Utilizing resources (multiple clusters) available on the Grid can make this task less time consuming for researcher.

Target application• High Energy Physics• Discrimination between signal and

background events coming from the particle detector (simulation)

• ROOT and Athena as basic data analysis tools

Observation Training of neural networks on the Grid requires many repeated tasks:

• job preparation, • submission, • monitoring of status,• gathering results.

Performing them manually is time consuming for the researcher→ Preparation of tools for automating such tasks can facilitate the whole process considerably.

Our Goals • Develop the tools facilitating usage of Grid for

multiple classification experiments• Investigate and validate algorithms for

distributed neural network training• Allow seamless integration with data analysis

tools such as ROOT

Node i

Update Weights

Calculate error

Node 2

Update Weights

Calculate error

Node 1

Update Weights

Calculate error

Read the training data& the network details

Distribute data to the nodes

Master

repeat untilstopping criterian

is metBackpropagation algoritm

Node i

Update Weights

Calculate error

Node i

Update Weights

Calculate error

Node 2

Update Weights

Calculate error

Node 2

Update Weights

Calculate error

Node 1

Update Weights

Calculate error

Node 1

Update Weights

Calculate error

Read the training data& the network details

Distribute data to the nodes

Master

repeat untilstopping criterian

is metBackpropagation algoritm

Jacobian

Hessjan

Node i

Jacobian

Hessjan

Node 2

Jacobian

Hessjan

Node 1

Read the training data& the network details

Distribute data to the nodes

Update Weights

Master

repeat untilstopping criterian

is met

Levenberg-Marquardt algoritm

Jacobian

Hessjan

Node i

Jacobian

Hessjan

Node 2

Jacobian

Hessjan

Node 1

Read the training data& the network details

Distribute data to the nodes

Update Weights

Master

repeat untilstopping criterian

is met

Jacobian

Hessjan

Node i

Jacobian

Hessjan

Node i

Jacobian

Hessjan

Node 2

Jacobian

Hessjan

Node 2

Jacobian

Hessjan

Node 1

Jacobian

Hessjan

Node 1

Read the training data& the network details

Distribute data to the nodes

Update Weights

Master

repeat untilstopping criterian

is met

Levenberg-Marquardt algoritm

Grid

User

Submit()

Cluster C

Cluster B

Cluster A

Grid

User

Submit()

Cluster CCluster C

Cluster BCluster B

Cluster ACluster A

11 March 2004Getting Ready for the Grid SAM: Tevatron Experiments Using the Grid CDF and D0 Need the Grid –Requirements, the CAF and SAM –Grid from the

Design of experiments on neural network's training for nonlinear

Ligand Pose Optimization with Atomic Grid-Based Convolutional Neural Networks · 2017-10-23 · Ligand Pose Optimization with Atomic Grid-Based Convolutional Neural Networks Matthew

Grid Job, Information and Data Management for the Run II Experiments at FNAL

Grid Experiments Advances May 23 rd 2008

Neural Turing Machine - Washington Stateeecs.wsu.edu/~cook/aiseminar/papers/steve.pdf · Neural Turing Machine - Experiments •Goal: to demonstrate NTM is ... Rumelhart, David E.;

Demand Response Experiments in Smart Grid … Research Institute of Electric Power Industry Demand Response Experiments in Smart Grid Projects in Japan Toru Hattori Socio-economic

Convolutional Neural Networksninamf/courses/315sp19/lectures/4_1-DL.pdfConvolutional neural networks •A specialized kind of neural network for processing data that has a known grid-like

EXPERIMENTS ON THE NEURAL CREST OF THE LAMPREY EMBRYOjeb.biologists.org/content/jexbio/28/3/247.full.pdf · EXPERIMENTS ON THE NEURAL CREST OF THE LAMPREY EMBRYO ... shallow sinks

Proposed experiments to prove the grid dimensions theory

February 12, 2000: Towards a US and LHC Grid Environment for Experiments Harvey Newman (CIT) Towards a US (and LHC) Grid Environment for HENP Experiments

Experiments with Neural Networks using R - IFDOseymour/R/neural.pdf · Experiments with Neural Networks using R Seymour Shlien December 15, 2016 1 Introduction Neural networks have

Grid cells: The position code, neural network models of ...clm.utexas.edu/fietelab/Papers/Review_gridcells_hippocampus08.pdf · Grid Cells: The Position Code, Neural Network Models

The Applications of Artificial Neural Networks to …...Artificial Neural Networks - Industrial and Control Engineering Applications 310 is crucial and the design of experiments has

Categorical Data Clustering Using Statistical Methods and Neural …petra/slides/syrcodis.pdf · 2009. 4. 12. · Introduction Clustering Statistical methods Neural Networks Experiments

Business Experiments in GRID

Grid Data Management - ut · Grid Data Management tools. Basics of Grid and Cloud computing 3/33 ... DNA experiments Radio telescopes Sensor networks Digitalizing data: books, documents,

Hexagonal Grid Image Processing Algorithms in Neural

Routine-Basis Experiments in PRAGMA Grid Testbed Yusuke Tanimura [email protected] Grid Technology Research Center National Institute of AIST

OCCF – The Realtime Grid. 1 Characteristics of Current Grid Computing Static data sets - Generally from fixed length experiments - Statistical measurements

Document Context Neural Machine Translation with Memory ... · Document Context Neural Machine Translation with Memory Networks Experiments and Analysis Experimental Setup Training/dev/test

Task 4 grid experiments

Replicability of Neural Computing Experiments

PPDGLHC Computing ReviewNovember 15, 2000 PPDG The Particle Physics Data Grid Making today’s Grid software work for HENP experiments, Driving GRID science

23.06.2008 Grid Experiments Advances May 23 rd 2008 Torsten Eymann Tina Balke Stefan König

Ian C. Smith ULGrid – Experiments in providing a campus grid

Connectivity and synchronization: comparison of neural ... Review1.pdf · Connectivity and synchronization: comparison of neural observations and experiments with toy networks Tokyo

Experiments with Neural Networks for Real Time ... · Experiments with Neural Networks for Real Time Implementation of Control 975 cannot exceed the physical capacity of that link

Experiments with Neural Networks for Real Time Implementation …papers.nips.cc/paper/1162-experiments-with-neural... · 2014-04-14 · Experiments with Neural Networks for Real Time

Building the PRAGMA Grid Through Routine-basis Experiments Cindy Zheng Pacific Rim Application and Grid Middleware Assembly San Diego Supercomputer Center

Oscillating-grid experiments in water and superfluid helium

Speech Recognition Using Demi-Syllable Neural Prediction …...Speech Recognition Using Demi-Syllable Neural Prediction Model 233 The results for 5000 word recognition experiments

SAM: Tevatron Experiments Using the Grid

Grid Computing in High Energy Physics Experiments

Hexagonal Grid Image Processing Algorithms in Neural Vision …paginas.fe.up.pt/~tavares/downloads/publications/relatorios/bio... · Hexagonal Grid Image Processing Algorithms in