randomized accuracy aware program transformations for efficient approximate computations
DESCRIPTION
Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations. Sasa Misailovic Joint work with. Zeyuan Allen Zhu. Jonathan Kelner. Martin Rinard. MIT CSAIL. …. …. …. …. …. Nodes represent computation Edges represent flow of data. …. …. …. …. …. - PowerPoint PPT PresentationTRANSCRIPT
Randomized Accuracy Aware Program Transformations for Efficient Approximate Computations
Sasa MisailovicJoint work with
Zeyuan Allen Zhu Jonathan Kelner Martin Rinard
MIT CSAIL
…
… … … …
• Nodes represent computation• Edges represent flow of data
…
• Functions – process individual data• Reduction nodes – aggregate data
… … … …
…
min
avgavg avgavg
• Functions – process individual data• Reduction nodes – aggregate data
… … … …
…
min
avgavg avgavg
Function substitution• Multiple implementations• Each has expected error/time
f2 f3f1
… … … …
…
min
avgavg avgavg
Function substitution• Multiple implementations• Each has expected error/time
… … … …
Sampling inputs of reduction nodes• Reductions consume fewer inputs
…
min
avgavg avgavg
… … … …
…
min
avgavg
Sampling inputs of reduction nodes• Reductions consume fewer inputs
… … … …
Tradeoff Space
Error
Time
Tradeoff Space
Error
Time
Optimal Tradeoff Curve
Error
Time Using the tradeoff curve:• Minimize time subject to error bound• Minimize error subject to time bound
Our Result
Analysis Optimization
Original program
Error bound
Optimizedprogram
Transformations
• Randomized computation• Guaranteed expected error/time tradeoff• -approximation of optimal tradeoff
Outline
Model of Computation
Tradeoff Curve Construction
Optimized Program Selection
Related Work
Model of Computation
…
f f
t
u v
w
t
u v
w
ff ggg g… … … …
min
avg avg avg avg
Model of Computation
…
f f
t
u v
w
t
u v
w
ff ggg g… … … …
min
avg avg avg avg
min
1
n
n n
f g
t
u v
w
avg avgm m
Structure of Computation
• Computation nodes DAGs of functions Functions: arbitrary code Process individual inputs
• Reduction nodes Aggregation functions Average, min, max, sum…
• Computation Tree Computation nodes and
reduction nodesmin
1
n
n n
f g
t
u v
w
avg avgm m
min
Accuracy-Aware Transformations
Function substitution• Multiple versions • Execute with probability • Each has error/time spec
Reduction sampling• Consume inputs
Probability of selecting eachinput:
• Derived error/time specifications
Average:
Min/max: 1
n
n n
avg avgm m
min
Accuracy-Aware Transformations
Function substitution• Multiple versions • Execute with probability • Each has error/time spec
Reduction sampling• Consume inputs
Probability of selecting eachinput:
• Derived error/time specifications
Average:
Min/max: 1
n
n n
avg avgm m…
avg
min
Accuracy-Aware Transformations
Function substitution• Multiple versions • Execute with probability • Each has error/time spec
Reduction sampling• Consume inputs
Probability of selecting eachinput:
• Derived error/time specifications
Average:
Min/max: 1
n
n n
avg avgm m…
avg
min
Accuracy-Aware Transformations
Function substitution• Multiple versions • Execute with probability • Each has error/time spec
Reduction sampling• Consume inputs
Probability of selecting eachinput:
• Derived error/time specifications
Average:
Min/max: 1
n
n n
avg avgm m…
avg
min
Program Configuration Vector
Defines transformed program • Functions: probability of
executing each version• Reductions: number of
elements to sample
1
n
n n
avg avgm m
…0.3 0.6 0.1 …
min
Configuration Vector
• Specifies program version• Functions: probability of
executing each version• Reductions: number of
elements to sample
1
n
n n
avg avgm m
…0.3 0.6 0.1 …
Find optimal program=
Find configuration vector that achieves optimal accuracy vs. performance tradeoff
min
1
n
n n
avg avgm m
Divide and conquer• For each subcomputation
construct tradeoff curve• Dynamic programming
Properties• Polynomial time• -approximation of
true tradeoff curve
Tradeoff Curve Construction: Algorithm
min
Tradeoff Curve Construction: Algorithm
1
n
n n
avg avgm m
Tradeoff Curve Construction: Algorithm
1
m m
n
n n
min
avg avg
Tradeoff Curve Construction: Algorithm
1
m m
n
n n
min
avg avg
Tradeoff Curve Construction: Algorithm
1
n
n n
min
avgm
Tradeoff Curve Construction: Algorithm
1
n
n n
min
Tradeoff Curve Construction: Algorithm
1
n
n n
min
Tradeoff Curve Construction: Algorithm
1
n
n n
min
Tradeoff Curve Construction: Algorithm
1
n
min
Tradeoff Curve Construction: Algorithm
Computation Node Optimization
Variables: • Probability to execute each version of • Range: • Sum:
(E2,T2)(E0,T0) (E1,T1)
Linear program
Computation Node Optimization
Variables: • Probability to execute each version of • Range: • Sum:
Objective:
Constraint:(E2,T2)(E0,T0) (E1,T1)
Linear program
Computation Node Optimization
Variables: • Probability to execute each version of • Range: • Sum:
Objective:
Constraint:(E2,T2)(E0,T0) (E1,T1)
Linear program
Computation Node Optimization
Variables: • Probability to execute each version of • Range: • Sum:
Objective:
Constraint:(E2,T2)(E0,T0) (E1,T1)
Linear program
The Algorithm: Reduction Nodes
Given error bound , find number of elements to sample
m
avg
The Algorithm: Reduction Nodes
Given error bound , find number of elements to sample
s.t.
m
avg
(𝐸𝑠𝑢𝑏 ,𝑇 𝑠𝑢𝑏)
The Algorithm: Reduction Nodes
Given error bound , find number of elements to sample
s.t.
From approximate tradeoff curve:
for
m
avg
(𝐸𝑠𝑢𝑏 ,𝑇 𝑠𝑢𝑏)
The Algorithm: Reduction Nodes
Given error bound , find number of elements to sample
m
avg
(𝐸𝑠𝑢𝑏 ,𝑇 𝑠𝑢𝑏)
s.t.
• Univariate optimization problem• Analogously, minimize error subject to
Approximate Tradeoff Curve
Error
Time
Bidimensional Discretization
Take elements at regular intervals
Approximating Tradeoff Curve
Error
Time
Bidimensional Discretization
Approximating Tradeoff Curve
Error
Time
Approximating Tradeoff Curve
Randomized configuration:• Execute with probability • Execute with probability
Error
Time
𝜅 1−𝜅
𝐴
𝐵
𝐶
Approximating Tradeoff Curve
Error
Time
�̂�𝐸𝜀𝐸
𝑇
Approximating Tradeoff Curve
Error
Time
𝑇
�̂�
𝑇
T
Properties of the Algorithm
Performance• Number of tradeoff curve points: • Most expensive operation: bidimensional discretization
Calling LP solver times Each call can have variables
Precision• Precision decreases linearly with the number of nodes • To obtain -approximation set intermediate
Space• Storing tradeoff curves:
Obtaining Optimized Programs
Tradeoff curves for all subcomputations:• Each curve contains partial configuration
Probability of executing local function nodes Number of inputs to sample from reduction node Error tolerated by subcomputation
• Distribution over optimal program configurations
Incrementally construct configuration vector:• For every execution• Traverse the tree, starting from root• Time to get full vector:
Related WorkAccuracy-aware transformations• Empirical justification: training/test input set
[Rinard ICS ‘06, Rinard OOPSLA ’07, Ansel et al. PLDI ‘09, Misailovic et al. ICSE ’10, Baek & Chilimbi PLDI ‘10 Hoffmann et al. ASPLOS ‘11, Sidiroglou et al. FSE ‘11]
• Probabilistic accuracy analysis for loop perforation [Misailovic et al. SAS ‘11, Chaudhuri et al. FSE ‘11]
Related WorkAccuracy-aware transformations• Empirical justification: training/test input set
[Rinard ICS ‘06, Rinard OOPSLA ’07, Ansel et al. PLDI ‘09, Misailovic et al. ICSE ’10, Baek & Chilimbi PLDI ‘10 Hoffmann et al. ASPLOS ‘11, Sidiroglou et al. FSE ‘11]
• Probabilistic accuracy analysis for loop perforation [Misailovic et al. SAS ‘11, Chaudhuri et al. FSE ‘11]
Ensuring safety of transformed programs• Separating critical and approximate parts of program
[Carbin & Rinard ISSTA ’10, Sampson et al. PLDI ’11]
• Verifying relaxed semantics of programs [Carbin et al. CSAIL-TR ‘11]
Analytic properties of programs [Majumdar & Saha RTSS ‘09, Chaudhuri et al. POPL’10, Ivancic et al. MEMOCODE ‘10 , Reed & Pierce ICFP ’10, Chaudhuri & Solar-Lezama PLDI ’10 , Chaudhuri et al. FSE ’11]
Summary
Model of Computation• Accuracy-aware program transformations• Effects on overall accuracy and execution time
Explore and Exploit Optimal Tradeoffs• Approximate optimal tradeoff curve construction• Polynomial, dynamic programming algorithm• Randomized program configurations to achieve tradeoffs
Envisioned Applications• Image and video processing, numerical algorithms,
queries on big data sets, machine learning, …• Optimization, fault tolerance, dynamic adaptation
