a grid research toolbox
DESCRIPTION
The Failure Trace Archive. and Cloud. A Grid Research Toolbox. DGSim. A. Iosup , O. Sonmez, N. Yigitbasi, H. Mohamed, S. Anoep, D.H.J. Epema. M. Jan. LRI/INRIA Futurs Paris, INRIA. PDS Group, ST/EWI, TU Delft. H. Li, L. Wolters. I. Raicu, C. Dumitrescu, I. Foster. LIACS, U. Leiden. - PowerPoint PPT PresentationTRANSCRIPT
April 19, 20231
A Grid Research Toolbox
Paris, France
A. Iosup, O. Sonmez, N. Yigitbasi, H. Mohamed, S. Anoep, D.H.J. EpemaPDS Group, ST/EWI, TU Delft
M. JanLRI/INRIA Futurs Paris, INRIA
DGSim
H. Li, L. WoltersLIACS, U. Leiden
I. Raicu, C. Dumitrescu, I. FosterU. Chicago
and Cloud
The FailureTraceArchive
April 19, 20232
A Layered View of the Grid World
• Layer 1: Hardware + OS• Automated• Non-grid (XtreemOS?)
• Layers 2-4: Grid Middleware Stack
• Low Level: file transfers, local resource allocation, etc.• High Level: grid scheduling• Very High Level: application environments (e.g., distributed objects)• Automated/user control• Simple to complex
• Layer 5: Grid Applications• User control• Simple to complex
HW + OS
Grid Low Level MW
Grid High Level MW
Grid Very High Level MW
Grid Applications
Gri
d M
W S
tack
April 19, 20233
Grid Work: Science or Engineering?
• Work on Grid Middleware and Applications
• When is work in grid computing science?• Studying systems to uncover their hidden laws• Designing innovative systems• Proposing novel algorithms• Methodological aspects:
repeatable experiments to verify and extend hypotheses
• When is work in grid computing engineering?• Showing that the system works in a common case, or
in a special case of great importance (e.g., weather prediction)
• When our students can do it (H. Casanova’s argument)
April 19, 20234
Grid Research Problem: We Are Missing Both Data and Tools
• Lack of data• Infrastructure
• number and type of resources, resource availability and failures
• Workloads• arrival process, resource consumption
• …
• Lack of tools• Simulators
• SimGrid, GridSim, MicroGrid, GangSim, OptorGrid, MONARC, …
• Testing tools that operate in real environments• DiPerF, QUAKE/FAIL-FCI
• …
We have problems to solve We have problems to solve in grid computing (as a in grid computing (as a sciencescience)!)!
April 19, 20235
Anecdote: Grids are far from being reliable job execution environments
• 99.999% reliableSmall Cluster
• 5x decrease in failure rate after first year [Schroeder and Gibson, DSN‘06]
Production Cluster
• >10% jobs fail [Iosup et al., CCGrid’06]DAS-2
• 99.99999% reliableServer
• 20-45% failures [Khalili et al., Grid’06]TeraGrid
• 27% failures, 5-10 retries [Dumitrescu et al., GCC’05]Grid3
CERN LCG jobsCERN LCG jobs74.71% successful74.71% successful
25.29% unsuccessful25.29% unsuccessful
Source: dboard-gr.cern.ch, May’07.
So at the moment our students cannot So at the moment our students cannot work in grid computing engineering!work in grid computing engineering!
April 19, 20236
The Anecdote at Scale
• NMI Build-and-Test Environment at U.Wisc.-Madison: 112 hosts, >40 platforms (e.g., X86-32/Solaris/5, X86-64/RH/9)
• Serves >50 grid middleware packages: Condor, Globus, VDT, gLite, GridFTP, RLS, NWS, INCA(-2), APST, NINF-G, BOINC …
Two years of functionality tests (‘04-‘06): Two years of functionality tests (‘04-‘06): over 1:3 runs have at least one failure!over 1:3 runs have at least one failure!
(1)(1) Test or perish!Test or perish!(2)(2) In today’s grids, reliability is In today’s grids, reliability is more important than performance!more important than performance!
A. Iosup, D.H.J.Epema, P. Couvares, A. Karp, M. Livny, Build-and-Test Workloads for Grid Middleware: Problem, Analysis, and Applications, CCGrid, 2007.
April 19, 20237
A Grid Research Toolbox
• Hypothesis: (a) is better than (b).
DGSim
1
2
3
For scenario 1, …
April 19, 20238
Research Questions
Q1: How to exchange grid/cloud data?Q1: How to exchange grid/cloud data?(e.g., Grid/Cloud * Archive)(e.g., Grid/Cloud * Archive)
Q2: What are the characteristics of grids/clouds?Q2: What are the characteristics of grids/clouds? (e.g., infrastructure, workload) (e.g., infrastructure, workload)
Q3: How to test and evaluate grids/clouds?Q3: How to test and evaluate grids/clouds?
April 19, 20239
Outline
1. Introduction and Motivation2. Q1: Exchange Data
1. The Grid Workloads Archive2. The Failure Trace Archive3. The Cloud Workloads Archive (?)
3. Q2: System Characteristics1. Grid Workloads2. Grid Infrastructure
4. Q3: System Testing and Evaluation
Traces in Distributed Systems Research• “My system/method/algorithm is better than yours
(on my carefully crafted workload)” • Unrealistic (trivial): Prove that “prioritize jobs from
users whose name starts with A” is a good scheduling policy
• Realistic? “85% jobs are short”; “10% Writes”; ...• Major problem in Computer Systems research
• Workload Trace = recording of real activity from a (real) system, often as a sequence of jobs / requests submitted by users for execution• Main use: compare and cross-validate new job and
resource management techniques and algorithms• Major problem: real workload traces from several
sourcesAugust 26, 2010
10
April 19, 202311
2.1. The Grid Workloads Archive [1/3]Content
6 traces online
http://gwa.ewi.tudelft.nlhttp://gwa.ewi.tudelft.nl
1.5 yrs >750K >250
A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters, D. Epema, The Grid Workloads Archive, FGCS 24, 672—686, 2008.
April 19, 202312
2.1. The Grid Workloads Archive [2/3]Approach: Standard Data Format (GWF)• Goals
• Provide a unitary format for Grid workloads;• Same format in plain text and relational DB (SQLite/SQL92);• To ease adoption, base on the Parallel Workloads Format
(SWF).
• Existing• Identification data: Job/User/Group/Application ID• Time and Status: Sub/Start/Finish Time, Job Status and Exit
code• Request vs. consumption: CPU/Wallclock/Mem
• Added• Job submission site• Job structure: bag-of-tasks, workflows• Extensions: co-allocation, reservations, others possible
A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters, D. Epema, The Grid Workloads Archive, FGCS 24, 672—686, 2008.
April 19, 202313
2.1. The Grid Workloads Archive [3/3]Approach: GWF Example
SubmitWait[s]Run#CPUs Mem [KB]Used Req
#CPUs
A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters, D. Epema, The Grid Workloads Archive, FGCS 24, 672—686, 2008.
April 19, 202314
2.2. The Failure Trace Archive
PresentationTypes of
systems• (Desktop) Grids• DNS servers• HPC Clusters• P2P systems
Stats• 25 traces• 100,000 nodes• Decades of
operation
The FailureTraceArchive
http://fta.inria.frhttp://fta.inria.fr
April 19, 202315
2.2. The Cloud Workloads Archive [1/2]
One Format Fits Them All
• Flat format• Job and Tasks• Summary (20 unique data fields) and Detail (60 fields)
• Categories of information• Shared with GWA, PWA: Time, Disk, Memory, Net• Jobs/Tasks that change resource consumption profile• MapReduce-specific (two-thirds data fields)
15
A. Iosup, R. Griffith, A. Konwinski, M. Zaharia, A. Ghodsi, I. Stoica, Data Format for the Cloud Workloads Archive, v.3, 13/07/10
CWJ CWJD CWT
CWTD
April 19, 202316
2.2. The Cloud Workloads Archive [2/2]
The Cloud Workloads Archive• Looking for invariants• Wr [%] ~40% Total IO, but absolute values
vary
• # Tasks/Job, ratio M:(M+R) Tasks, vary• Understanding workload evolution
Trace ID Total IO [MB]
Rd. [MB] Wr [%] HDFS Wr[MB]
CWA-01 10,934 6,805 38% 1,538
CWA-02 75,546 47,539 37% 8,563
April 19, 202317
Outline
1. Introduction and Motivation2. Q1: Exchange Data
1. The Grid Workloads Archive2. The Failure Trace Archive3. The Cloud Workloads Archive (?)
3. Q2: System Characteristics1. Grid Workloads2. Grid Infrastructure
4. Q3: System Testing and Evaluation
April 19, 202318
3.1. Grid Workloads [1/7]
Analysis Summary: Grid workloads different, e.g., from parallel production envs. (HPC)• Traces: LCG, Grid3, TeraGrid, and DAS
• long traces (6+ months), active environments (500+K jobs per trace, 100s of users), >4 million jobs
• Analysis• System-wide, VO, group, user characteristics• Environment, user evolution• System performance
• Selected findings• Almost no parallel jobs• Top 2-5 groups/users dominate the workloads• Performance problems: high job wait time, high failure rates
A. Iosup, C. Dumitrescu, D.H.J. Epema, H. Li, L. Wolters, How are Real Grids Used? The Analysis of Four Grid Traces and Its Implications, Grid 2006.
April 19, 202319
3.1. Grid Workloads [2/7]Analysis Summary: Grids vs. Parallel Production Systems• Similar CPUTime/Year, 5x larger arrival bursts
GridsParallel Production
Environments (Large clusters,
supercomputers)
LCG clusterdaily peak: 22.5k jobs
A. Iosup, D.H.J. Epema, C. Franke, A. Papaspyrou, L. Schley, B. Song, R. Yahyapour, On Grid Performance Evaluation using Synthetic Workloads, JSSPP’06.
April 19, 202320
Bags-of-Tasks (BoTs)
3.1. Grid Workloads [3/7]More Analysis: Special Workload Components
BoT = set of jobs…
…that start at most Δs after the first job
Time [units]
Parameter Sweep App. = BoT with same binary
Workflows (WFs)
WF = set of jobs with precedence(think Direct Acyclic Graph)
April 19, 202321
• Selected Findings• Batches predominant in grid workloads; up to 96%
CPUTime
• Average batch size (Δ≤120s) is 15-30 (500 max)• 75% of the batches are sized 20 jobs or less
3.1. Grid Workloads [4/7]
BoTs are predominant in grids
A. Iosup, M. Jan, O. Sonmez, and D.H.J. Epema, The Characteristics and Performance of Groups of Jobs in Grids, Euro-Par, LNCS, vol.4641, pp. 382-393, 2007.
Grid’5000 NorduGrid GLOW (Condor)
Submissions
26k 50k 13k
Jobs 808k (951k)
738k (781k) 205k (216k)
CPU time 193y (651y)
2192y (2443y)
53y (55y)
April 19, 202322
• Traces
• Selected Findings
• Loose coupling• Graph with 3-4 levels• Average WF size is 30/44 jobs• 75%+ WFs are sized 40 jobs or less, 95% are sized 200 jobs
or less
3.1. Grid Workloads [5/7]
Workflows exist, but they seem small
S. Ostermann, A. Iosup, R. Prodan, D.H.J. Epema, and T. Fahringer. On the Characteristics of Grid Workflows, CoreGRID Integrated Research in Grid Computing (CGIW), 2008.
April 19, 202323
• Adapted to grids: percentage parallel jobs, other values.• Validated with 4 grid and 7 parallel production env. traces
3.1. Grid Workloads [6/7]
Modeling Grid Workloads: Feitelson adapted
A. Iosup, D.H.J. Epema, T. Tannenbaum, M. Farrellee, and M. Livny. Inter-Operating Grids Through Delegated MatchMaking, ACM/IEEE Conference on High Performance Networking and Computing (SC), pp. 13-21, 2007.
April 19, 202324
• Single arrival process for both BoTs and parallel jobs• Reduce over-fitting and complexity of “Feitelson adapted”
by removing the RunTime-Parallelism correlated model• Validated with 7 grid workloads
3.1. Grid Workloads [7/7]
Modeling Grid Workloads: adding users, BoTs
A. Iosup, O. Sonmez, S. Anoep, and D.H.J. Epema. The Performance of Bags-of-Tasks in Large-Scale Distributed Systems, HPDC, pp. 97-108, 2008.
April 19, 202325
3.2. Grid Infrastructure [1/5]
Existing resource models and data
• Compute Resources• Commodity clusters [Kee et al., SC’04]
• Desktop grids resource availability [Kondo et al., FCFS’07]
• Network Resources• Structural generators: GT-ITM [Zegura et al., 1997]
• Degree-based generators: BRITE [Medina et al., 2001]
• Storage Resources, other resources• ?
Source: H. CasanovaStatic! Static!
Resource dynamic, evolution, … Resource dynamic, evolution, … NOT consideredNOT considered
April 19, 202326
3.2. Grid Infrastructure [2/5]Resource dynamics in cluster-based grids• Environment: Grid’5000 traces
• jobs 05/2004-11/2006 (30 mo., 950K jobs)• resource availability traces 05/2005-11/2006 (18 mo., 600K
events)
• Resource availability model for multi-cluster grids
Grid-level availability: 70%
A. Iosup, M. Jan, O. Sonmez, and D.H.J. Epema, On the Dynamic Resource Availability in Grids, Grid 2007, Sep 2007.
April 19, 202327
• Correlated failure Maximal set of failures (ordered according to increasing event
time), of time parameter in which for any two successive failures E and F,
where returns the timestamp of the event; = 1-3600s.
3.2. Grid Infrastructure [3/5] Correlated Failures
• Grid-level view• Range: 1-339• Average: 11
• Cluster span• Range: 1-3• Average: 1.06• Failures “stay” within cluster Size of correlated failures
CD
F Average
Grid-level view
A. Iosup, M. Jan, O. Sonmez, and D.H.J. Epema, On the Dynamic Resource Availability in Grids, Grid 2007, Sep 2007.
April 19, 202328
• Assume no correlation of failure occurrence between clusters
• Which site/cluster? • fs, fraction of failures at cluster s
MTBF MTTR Correl.
• Weibull distribution for IAT• Shape parameter > 1: increasing hazard rate
the longer a node is online, the higher the chances that it will fail
3.2. Grid Infrastructure [4/5]
Dynamics Model
A. Iosup, M. Jan, O. Sonmez, and D.H.J. Epema, On the Dynamic Resource Availability in Grids, Grid 2007, Sep 2007.
April 19, 202329
3.2. Grid Infrastructure [5/5]
Evolution Model
A. Iosup, O. Sonmez, and D. Epema, DGSim: Comparing Grid Resource Management Architectures through Trace-Based Simulation, Euro-Par 2008.
April 19, 202330
• Grid workloads very different from those of other systems, e.g., parallel production envs. (large clusters, supercomputers)• Batches of jobs are predominant [Euro-Par’07,HPDC’08]• Almost no parallel jobs [Grid’06]• Workload model [SC’07, HPDC’08]• Clouds? (upcoming)
• Grid resources are not static• Resource dynamics model [Grid’07]• Resource evolution model [EuroPar’08]• Clouds? [CCGrid’11]
• Archives: easy to share traces and associated research
Q1,Q2: What are the characteristics of gridsQ1,Q2: What are the characteristics of grids (e.g., infrastructure, workload)? (e.g., infrastructure, workload)?
http://gwa.ewi.tudelft.nlhttp://gwa.ewi.tudelft.nl
April 19, 202331
Outline
1. Introduction and Motivation2. Q1: Exchange Data
1. The Grid Workloads Archive2. The Failure Trace Archive3. The Cloud Workloads Archive (?)
3. Q2: System Characteristics1. Grid Workloads2. Grid Infrastructure
4. Q3: System Testing and Evaluation
April 19, 202332
4.1. GrenchMark: Testing in LSDCSsAnalyzing, Testing, and Comparing Systems
• Use cases for automatically analyzing, testing, and comparing systems (or middleware)• Functionality testing and system tuning• Performance testing/analysis of applications• Reliability testing of middleware• …
• For grids and clouds, this problem is difficult !• Testing in real environments is difficult/costly/both• Grids/clouds change rapidly• Validity and reproducibility of tests• …
April 19, 202333
4.1. GrenchMark: Testing LSDCSs Architecture Overview
GrenchMark = Grid Benchmark
April 19, 202335
4.1. GrenchMark: Testing LSDCSs Testing a Large-Scale Environment (1/2)
• Testing a 1500-processors Condor environment• Workloads of 1000 jobs, grouped by 2, 10, 20, 50, 100, 200• Test finishes 1h after the last submission• Results
• >150,000 jobs submitted• >100,000 jobs successfully run, >2 yr CPU time in 1
week• 5% jobs failed (much less than other grids’
average) • 25% jobs did not start in time and where cancelled
April 19, 202336
4.1. GrenchMark: Testing LSDCSs Testing a Large-Scale Environment (2/2)• Performance metrics
system-, job-, operational-, application-, and service-level
April 19, 202337
4.1. GrenchMark: Testing in LSDCSs ServMark: Scalable GrenchMark
• Blending DiPerF and GrenchMark.
• Tackles two orthogonal issues: • Multi-sourced testing
(multi-user scenarios, scalability)• Generate and run dynamic test
workloads with complex structure(real-world scenarios, flexibility)
• Adds• Coordination and automation
layers• Fault tolerance module
DiPerF
GrenchMark
ServMark
April 19, 202338
Performance Evaluation of Clouds [1/3]
C-Meter: Cloud-Oriented GrenchMark
Yigitbasi et al.: C-Meter: A Framework for Performance Analysis of Computing Clouds. Proc. of CCGRID 2009
April 19, 202339
Performance Evaluation of Clouds [2/3]
Low Performance for Sci.Comp.
• Evaluated the performance of resources from four production, commercial clouds. • GrenchMark for evaluating the performance of cloud
resources• C-Meter for complex workloads
• Four production, commercial IaaS clouds: Amazon Elastic Compute Cloud (EC2), Mosso, Elastic Hosts, and GoGrid.
• Finding: cloud performance low for sci.comp.
S. Ostermann et al., A Performance Analysis of EC2 Cloud Computing Services for Scientific Computing, Cloudcomp 2009, LNICST 34, pp.115–131, 2010.
A. Iosup et al.,Performance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing, IEEE TPDS, vol.22(6), 2011.
April 19, 202340
Performance Evaluation of Clouds [3/3]
Cloud Performance Variability• Long-term performance variability of production cloud
services• IaaS:
Amazon Web Services• PaaS:
Google App Engine
• Year-long performance information for nine services• Finding: about half of the cloud services
investigated in this work exhibits yearly and daily patterns; impact of performance variability depends on application.A. Iosup, N. Yigitbasi, and D. Epema, On the Performance Variability of Production Cloud Services, CCGrid 2011.
Amazon S3: GET US HI operations
April 19, 202341
4.2. DGSim: Simulating Multi-Cluster GridsGoal and Challenges
• Simulate various grid resource management architectures• Multi-cluster grids• Grids of grids (THE grid)
• Challenges• Many types of architectures• Generating and replaying grid workloads• Management of simulations
• Many repetitions of a simulation for statistical relevance• Simulations with many parameters• Managing results (e.g., analysis tools)• Enabling collaborative experiments
Two GRM architectures
DGSim
April 19, 202342
4.2. DGSim: Simulating Multi-Cluster GridsOverview
Discrete-EventSimulator
DGSim
April 19, 202343
4.2. DGSim: Simulating Multi-Cluster GridsSimulated Architectures (Sep 2007)
Hybrid hierarchical/ decentralize
d
Decentralized
Hierarchical
Independent
Centralized
DGSimA. Iosup, D.H.J.Epema, T. Tannenbaum, M. Farrellee, M. Livny, Inter-Operating Grids through Delegated MatchMaking, SC, 2007.
April 19, 202344
Q3: How to test and evaluate grids/clouds?Q3: How to test and evaluate grids/clouds?
• GrenchMark+C-Meter: testing large-scale distrib. sys.• Framework• Testing in real environments performance, reliability,
functionality• Uniform process: metrics, workloads• Real tool available
• DGSim: simulating multi-cluster grids • Many types of architectures• Generating and replaying grid workloads• Management of the simulations
grenchmark.st.ewi.tudelft.nlgrenchmark.st.ewi.tudelft.nl
dev.globus.org/wiki/Incubator/ServMarkdev.globus.org/wiki/Incubator/ServMark
April 19, 202345
Take Home Message: Research ToolboxTake Home Message: Research Toolbox
• Understanding how real systems work• Modeling workloads and infrastructure• Compare grids and clouds with other platforms (parallel production
env.,…)
• The Archives: easy to share system traces and associated research • Grid Workloads Archive• Failure Trace Archive• Cloud Workloads Archive (upcoming)
• Testing/Evaluating Grids/Clouds• GrenchMark• ServMark: Scalable GrenchMark• C-Meter: Cloud-oriented GrenchMark• DGSim: Simulating Grids (and Clouds?)
Publications
2006: Grid, CCGrid, JSSPP
2007: SC, Grid, CCGrid, …
2008: HPDC, SC, Grid, …
2009: HPDC, CCGrid, …
2010: HPDC, CCGrid (Best Paper
Award), EuroPar, …
2011: IEEE TPDS, IEEE Internet
Computing, CCGrid, …
April 19, 202346
Thank you for your attention! Questions? Suggestions? Observations?
Alexandru Iosup
[email protected]://www.pds.ewi.tudelft.nl/~iosup/ (or google “iosup”)Parallel and Distributed Systems GroupDelft University of Technology
- http://www.st.ewi.tudelft.nl/~iosup/research.html
- http://www.st.ewi.tudelft.nl/~iosup/research_gaming.html
- http://www.st.ewi.tudelft.nl/~iosup/research_cloud.html
More Info:
Do not hesitate to contact me…