cardio: cost-aware replication for data-intensive workflows
DESCRIPTION
CARDIO: Cost-Aware Replication for Data-Intensive workflOws. Presented by Chen He. Motivation. Is large scale cluster reliable? 5 average worker deaths per Map-Reduce job At least 1 disk failure in every run of a 6- hour MapReduce job on a 4000-node cluster. Motivation. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/1.jpg)
CARDIO: Cost-Aware Replication for Data-Intensive workflOws
Presented by Chen He
![Page 2: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/2.jpg)
Motivation• Is large scale cluster reliable?
5 average worker deaths per Map-Reduce job At least 1 disk failure in every run of a 6- hour MapReduce job on a 4000-node cluster
![Page 3: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/3.jpg)
Motivation
• How to prevent node failure from affecting performance?– Replication
• Capacity constraint• Replication time, etc
– Regeneration through re-execution• Delay program progress• Cascaded re-execution
![Page 4: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/4.jpg)
Motivation
AVAILABILITY
COST
All pictures adopted from the Internet
![Page 5: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/5.jpg)
Outline
• Problem Exploration• CARDIO Model• Hadoop CARDIO System• Evaluation• Discussion
![Page 6: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/6.jpg)
Problem Exploration
• Performance Costs– Replication cost (R)– Regeneration cost (G)– Reliability cost (Z)– Execution cost (A)– Total cost (T)– Disk cost (Y)
T=A+Z Z=R+G
![Page 7: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/7.jpg)
Problem Exploration
• Experiment Environment– Hadoop 0.20.2– 25 VMs– Workloads: Tagger->Join->Grep->RecordCounter
![Page 8: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/8.jpg)
Problem Exploration Summary
• Replication Factor for MR Stages
![Page 9: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/9.jpg)
Problem Exploration Summary
• Detailed Execution Time of 3 Cases
![Page 10: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/10.jpg)
CARDIO Model
• Block Failure Model– Output of stage i is – Replication factor is– Total block number is – Single block failure probability is– Failure probability in stage i:
( ) 1 (1 )i ix bif x p
ixiD
ib
p
![Page 11: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/11.jpg)
CARDIO Model
• Cost Computation Model– Total time of stage i:– Replication cost of stage i: – Expected regeneration time of stage i:
– Reliability cost for all stages:– Storage Constraint C of all stages:
– Choose to minimize Z
1i i i iT A R G
( 1)i i i iG f x T
1 2{ , ,... }nX x x x
1
1 1
n n
i ii i
Z R G
1
n
i ii
Y xY C
i i iR x Y
![Page 12: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/12.jpg)
CARDIO Model
• Dynamic Replication– Replication number x may vary during the
program approaching• Job is in Step k, the replication factor at this step is:
( ), 1, 2,... 1, 2,...,ix k i k k n
![Page 13: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/13.jpg)
CARDIO Model
• Model for Reliability– Minimize– Based on
– In the condition of
1 2{ ( ), ( ),... ( )}nX x k x k x k
1 2
( ) ( )n n
k k
Z R k G k
1
( ) ( )k
i ii
Y k x k Y C
![Page 14: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/14.jpg)
CARDIO Model
• Resource Utilization Model– Model Cost = resource utilized– Resource type Q
• CPU, Network, Disk, and Storage resource, etc.• Utilization of q resource in stage i:• Normalize usage by
• Relative costs weights:
, 1,2,...i qu q Q
,i q,
,
,1
, 1, 2,...i qi q n
j qj
uq Q
u
, 1, 2,...qw q Q
![Page 15: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/15.jpg)
CARDIO Model
• Resource Utilization Model– The cost for A is:
– Total Cost:
– Optimization target:• Choose to minimize T
,1
Q
i q i qq
A w
1' '
,1 1 1 1
Qn n n
q i q i ii q i i
T A Z w R G
1 2{ ( ), ( ),... ( )}nX x k x k x k
![Page 16: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/16.jpg)
CARDIO Model
• Optimization Problem– Job optimality (JO)– Stage optimality (SO)
![Page 17: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/17.jpg)
Hadoop CARDIO System
• CardioSense– Obtain progress from JT periodically– Be triggered by pre-configured threshold-value– Collect resource usage statistics for running stages– Rely on HMon on each worker node
• HMon based on Atop has low overhead
![Page 18: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/18.jpg)
Hadoop CARDIO System
• CardioSolve– Receive data from CardioSense– Solve SO problem– Decide the replication factors for current and
previous stages
![Page 19: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/19.jpg)
Hadoop CARDIO System
• CardioAct– Implement the command from CardioSolve– Use HDFS API setReplication(file, replicaNumber)
![Page 20: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/20.jpg)
Hadoop CARDIO System
![Page 21: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/21.jpg)
Evaluation• Several Important Parameters
– p is the failure rate 0.2 if not specified– is the time to replicate a data unit, 0.2 as well– is the computation resource of stage i, it follows
uniform distribution U(1,Cmax),Cmax=100 in general.– is the output of stage i, it is obtained from a
uniform distribution U(1, Dmax), Dmax varies within the [1,Cmax].
– C is the storage constraint for the whole process. Default value is
iC
iD
![Page 22: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/22.jpg)
Evaluation
• Effect of Dmax
![Page 23: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/23.jpg)
Evaluation
• Effect of Failure rate p
![Page 24: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/24.jpg)
Evaluation
• Effect of block size
![Page 25: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/25.jpg)
Evaluation
• Effect of different resource constraints++ means over-utilzed, and this type of resource is regarded as expensive
P=0.08, C=204GB, delta=0.6
S3 is CPU intensive
DSK has similar performance pattern as NET
CPU 0010, NET 0011, DSKIO 0011,STG0011
![Page 26: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/26.jpg)
Evaluation
S2 re-execute more frequently due to the failure injection. Because it has large data output.
P=0.02, 0.08 and 0.1 1 , 3, 21
API reason
![Page 27: CARDIO: Cost-Aware Replication for Data-Intensive workflOws](https://reader036.vdocument.in/reader036/viewer/2022081516/56814e5e550346895dbbfb8a/html5/thumbnails/27.jpg)
Discussion
• Problems– Typos and misleading
symbols– HDFS API setReplication()
• Any other ideas?