concurrency control for parallel machine learning
DESCRIPTION
"Concurrency Control for Parallel Machine Learning" presentation at AMPCamp 5 by Dimitris PapailiopoulosTRANSCRIPT
Concurrency Control for
Parallel Machine Learning
Dimitris Papailiopoulos
Xinghao Pan, Joseph Gonzalez, Stefanie Jegelka, Tamara Broderick, Dimitris Papailiopoulos, Joseph Bradley, Michael I. Jordan
Data
Model
State
Serial Inference
Model
State
Parallel Inference
Processor 1
Processor 2
Data
Data
Model
State
Parallel Inference
Processor 1
Processor 2
Concurrency:more machines = less time
Correctness:serial equivalence
?
Data
Model
State
Coordination Free Parallel
Inference
Processor 1
Processor 2
Ignore collisions
?
Concurrency:(almost) free
+
Speedup = #CPU
Correctness?Not always...
Correctness
Serial
Low High
Correctness
Concurrency
Coordination-
free
Serial
High
Low High
Low
Correctness
Concurrency
Coordination-
free
Serial
High
Low High
Low
Concurrency
ControlDatabase mechanismso Guarantee correctnesso Maximize concurrency Mutual exclusion Optimistic CC
Data
Model
State
Mutual Exclusion Through
Locking
Processor 1
Processor 2
Introduce locking (scheduling) protocols to prevent
conflicts.
Data
Model
State
Processor 1
Processor 2
✗
Enforce local serialization to avoid conflicts.
Mutual Exclusion Through
Locking
Data
Model
State
Optimistic Concurrency Control
Processor 1
Processor 2
Allow computation to proceed without blocking.
Kung & Robinson. On optimistic methods for concurrency
control.
ACM Transactions on Database Systems 1981
Data
Model
State
Optimistic Concurrency Control
Processor 1
Processor 2
✗ ✗
Validate potential conflicts.
Invalid Outcome
Kung & Robinson. On optimistic methods for concurrency
control.
ACM Transactions on Database Systems 1981
Data
Model
State
Optimistic Concurrency Control
Processor 1
Processor 2
✗ ✗Rollback and Redo
Take a compensating action.
Kung & Robinson. On optimistic methods for concurrency
control.
ACM Transactions on Database Systems 1981
Concurrency Control
14
Coordination Free:
Provably fast and correct under key assumptions.
Concurrency Control:
Provably correct and fast under key assumptions.
Systems Ideas to
Improve Efficiency
Clusteri
ng
Online
Facility
Location
Submodular
MaximizationSubset selection, diminishing
marginal gainsMax Graph
Cut
Set
Cover
Sensor Placement
Social Network
Influence
Propagation
Document
Summarization
Machine Learning + Concurrency
Control(Xinghao Pan et al.)
SportsFootball
Word Series
Giants
Cardinals
PoliticsMidterm
Obama
Democrat
Tea
FinanceQE
market
interest
Dow
Topic Modelling
Correlation
Clustering
Community
Detection
Deduplication
Clusteri
ng
Online
Facility
Location
Submodular
MaximizationSubset selection, diminishing
marginal gainsMax Graph
Cut
Set
Cover
Sensor Placement
Social Network
Influence
Propagation
Document
Summarization
SportsFootball
Word Series
Giants
Cardinals
PoliticsMidterm
Obama
Democrat
Tea
FinanceQE
market
interest
Dow
Topic Modelling
Correlation
Clustering
Community
Detection
Deduplication
Serial ML
algorithm
Sequence of
transactions
Identify potential
conflicts
Apply Concurrency
Control
mechanisms
Parallel ML
algorithm
Machine Learning + Concurrency
Control(Xinghao Pan et al.)
Application: Deduplication
Computer Science
Division – University of
California Berkeley CA
University of California at Berkeley
Department of
Physics Stanford
University California
Lawrence Berkeley National
Labs <ref>California</ref>
Computer Science
Division – University of
California Berkeley CA
University of California at Berkeley
Department of
Physics Stanford
University California
Lawrence Berkeley National
Labs <ref>California</ref>
Application: Deduplication
Serial Correlation Clustering
Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.
Journal of the ACM (JACM), 55(5):23, 2008.
Serially process vertices
Serial Correlation Clustering
Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.
Journal of the ACM (JACM), 55(5):23, 2008.
Serially process vertices
Serial Correlation Clustering
Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.
Journal of the ACM (JACM), 55(5):23, 2008.
Serially process vertices
Serial Correlation Clustering
Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.
Journal of the ACM (JACM), 55(5):23, 2008.
Serially process vertices
Serial Correlation Clustering
Nir Ailon, Moses Charikar, and Alantha Newman.Aggregating inconsistent information: ranking and clustering.
Journal of the ACM (JACM), 55(5):23, 2008.
Serially process vertices
Approximation 3 OPT (in expectation)
Parallel Correlation Clustering
Concurrency Control Correlation Clustering
(C4)Parallel Correlation ClusteringCannot introduce
adjacent cluster
centers
Resolve by
Mutual Exclusion
Concurrency Control Correlation Clustering
(C4)Common neighbor must be
assigned to earliest center
Resolve by
Optimistic Concurrency
Control
ResolutionAssign common neighbor to earliest cluster
Optimistic AssumptionNo other new cluster created
?
Properties of C4(Concurrency Control Correlation Clustering)
Theorem: C4 is correct.
C4 preserves same guarantees as serial algorithm (3
OPT).Co
rre
ctn
es
sC
on
cu
rren
cy
Theorem: C4 has provably small overheads.
= almost linear speedupExpected #blocked transactions < 2τ |E| / |V|.
τ ≡ diff in parallel cpu’s progress
Empirical Validation on Billion Edge
Graphs
Amazon EC2 r3.8xlarge instances
Multicore up to 16 threads
Real and synthetic graphs
100 runs (10 random orderings x 10 runs)
Graph Vertices Edges
IT-2004 Italian web-graph 41 Million 1.14 Billion
Webbase-2001 WebBase crawl 118 Million 1.02 Billion
Erdos-Renyi Synthetic random 100 Million ≈ 1.0 Billion
C4: Cost of Coordination
< 0.02% blocked
C4: Speed-up
Ideal
10x
speedu
p
ConclusionConcurrency Control
for Parallel ML
o Guarantee
correctness
o Maximize
concurrency
Code release in the works!
https://amplab.cs.berkeley.edu/projects/cc
ml/
ApplicationsCorrelation Clustering
Submodular Maximization
Clustering
Online Facility Location
Feature Modeling