su yuxin jan 20, 2014 petuum: an iterative-convergent distributed machine learning framework
TRANSCRIPT
![Page 1: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/1.jpg)
SU YUXINJAN 20, 2014
Petuum: An Iterative-Convergent Distributed
Machine Learning Framework
![Page 2: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/2.jpg)
Outline
Introduction
Implementation
Questions
Demo
![Page 3: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/3.jpg)
Introduction to Petuum
![Page 4: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/4.jpg)
Bulk Synchronous Parallel
![Page 5: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/5.jpg)
Asynchronous
Parameters read / update at any time
![Page 6: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/6.jpg)
Stale Synchronous Parallel
![Page 7: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/7.jpg)
Convergence
![Page 8: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/8.jpg)
Programming
read(table, row, col)
inc(table, row, col, value)
iteration()
![Page 9: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/9.jpg)
Implementation
![Page 10: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/10.jpg)
Overview in Logic
![Page 11: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/11.jpg)
Overview in the Real
![Page 12: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/12.jpg)
Main Components
![Page 13: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/13.jpg)
Table
![Page 14: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/14.jpg)
ConsistencyController::DoGet()
![Page 15: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/15.jpg)
ConsistencyController::iterate()
![Page 16: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/16.jpg)
Server::GetRow()
![Page 17: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/17.jpg)
Least-Recently-Used(LRU) Strategy
![Page 18: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/18.jpg)
Questions
![Page 19: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/19.jpg)
Is Lock-Free Possible ?
Data exchange in real-time ? next …
![Page 20: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/20.jpg)
Is Auto-Rescheduling Possible ?
sub-centralized server
reduce communication cost
![Page 21: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/21.jpg)
Is Auto-Partition Possible ?
Run ML algorithms like that in a single thread
A Solution for all ML algorithms
![Page 22: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/22.jpg)
In-Memory or In-Storage ?
Data capacity is greater than memory size.
Memory should be a cache for disk storage.
Solution for disk storage: Hadoop Spark ….
![Page 23: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/23.jpg)
New Schema to Reduce the Upper Bound?
![Page 24: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/24.jpg)
STRADS Scheduler
Variable Correlations Auto-Parallelization
Dynamic Prioritization Monitor the contribution of variables to objective
function
Load-Balancing in Task
![Page 25: SU YUXIN JAN 20, 2014 Petuum: An Iterative-Convergent Distributed Machine Learning Framework](https://reader030.vdocument.in/reader030/viewer/2022032611/56649e7a5503460f94b7ab00/html5/thumbnails/25.jpg)
Demo
Switch to my laptop …