prophesy: analysis and modeling of parallel and distributed applications

Prophesy: Analysis and Modeling of Parallel Prophesy: Analysis and Modeling of Parallel and Distributed Applicationsand Distributed Applications

Valerie TaylorValerie TaylorTexas A&M UniversityTexas A&M University

Seung-Hye Jang, Mieke Prajugo, Xingfu Wu – TAMUSeung-Hye Jang, Mieke Prajugo, Xingfu Wu – TAMUEwa Deelman – ISI Ewa Deelman – ISI

Juan Gilbert – Auburn UniversityJuan Gilbert – Auburn UniversityRick Stevens – Argonne National LaboratoryRick Stevens – Argonne National Laboratory

SPONSORS: NSF, NASASPONSORS: NSF, NASA

http://prophesy.cs.tamu.eduhttp://prophesy.cs.tamu.edu22

Performance ModelingPerformance Modeling

Necessary for good performanceNecessary for good performance Requires significant time and effortRequires significant time and effort

MD Code Throughput

0

1

2

3

4

1 9 25 49 81 121

Number of Processors

Tim

este

ps/s

Exper.

Theo.


OutlineOutlineOutlineOutline

Prophesy InfrastructureProphesy Infrastructure

Modeling TechniquesModeling Techniques

Case StudiesCase Studies

SummarySummary


Problem StatementProblem StatementProblem StatementProblem Statement

Given:Given:• Performance models and analyses are criticalPerformance models and analyses are critical

– Requires significant development timeRequires significant development time

• Parallel and distributed systems are complexParallel and distributed systems are complex GoalGoal

Efficient execution of parallel & distributed Efficient execution of parallel & distributed applicationsapplications

Proposed SolutionProposed Solution• Automate as much as possibleAutomate as much as possible• Community involvementCommunity involvement


Prophesy SystemProphesy SystemPROPHESY GUI

Profiling &

Instrument.

Actual

Execution

Performance Database

TemplateDatabase

SystemsDatabase

ModelBuilder

PerformancePredictor

DATACOLLECTION

DATABASES DATAANALYSIS


Automated InstrumentationAutomated Instrumentation In-line data collectionIn-line data collection Instrument at one of several pre-Instrument at one of several pre-

defined levelsdefined levels Allow for user-specified Allow for user-specified

instrumentationinstrumentation

Profiling &

Instrument.

Actual

ExecutionT=E * f;T=E * f;for (I=1; I<N; I++){for (I=1; I<N; I++){ V(I) = A(I) * C(I);V(I) = A(I) * C(I); B(I) = A(2I + 4);B(I) = A(2I + 4);}}

T=E * f;T=E * f;INSTRUMENTATION CODEINSTRUMENTATION CODEfor (I=1; I<N; I++){for (I=1; I<N; I++){ V(I) = A(I) * C(I);V(I) = A(I) * C(I); B(I) = A(2I + 4);B(I) = A(2I + 4);}}INSTRUMENTATION CODEINSTRUMENTATION CODE


DatabasesDatabases

Hierarchical organizationHierarchical organization Organized into 4 areas:Organized into 4 areas:

• ApplicationApplication• ExecutableExecutable• RunRun• Performance StatisticsPerformance Statistics

Performance Database

TemplateDatabase

SystemsDatabase


Prophesy DatabaseProphesy Database

Application Executable RunApplication

Performance

Modules

Function Performance

Basic Unit Performance

Data Structure Performance

Inputs

Systems

Resource Connection

Functions

Module_Info

Control Flow

Compilers

Model Template Function_Info

LibraryModel_Info


Data AnalysisData Analysis

Develop performance Develop performance modelsmodels

Make predictionsMake predictions Performance tune codesPerformance tune codes Identify best Identify best

implementation implementation Identify trendsIdentify trends

ModelBuilder

PerformancePredictor


Automated Modeling TechniquesAutomated Modeling Techniques Utilize information in the template and Utilize information in the template and

system databasessystem databases Currently include three techniquesCurrently include three techniques

• Curve fittingCurve fitting• ParameterizationParameterization• Composition using coupling valuesComposition using coupling values


Curve Fitting: UsageCurve Fitting: UsageCurve Fitting: UsageCurve Fitting: Usage

Application Performance


Basic Unit Performance

Data Structure

Performance

Model Template

Matrix-matrix multiply: Matrix-matrix multiply: LSF : 3LSF : 3

PerformancePerformanceDataData

Analytical EquationAnalytical Equation(Octave: LSF)(Octave: LSF)


Matrix-matrix multiplication, 16P, IBM SPMatrix-matrix multiplication, 16P, IBM SP


Parameterization: UsageParameterization: UsageParameterization: UsageParameterization: Usage

Model Template

Matrix-matrix multiply: Matrix-matrix multiply: Parameterization : Parameterization : Parameter(P, SGI Origin2000, N, ADDM, Parameter(P, SGI Origin2000, N, ADDM,

MPISR, MPIBC)MPISR, MPIBC)

Analytical EquationAnalytical Equation(Octave: Parameterization)(Octave: Parameterization)

Systems

Resource Connection

System Data:System Data:MPISR, MPIBC, ADDMMPISR, MPIBC, ADDM


Modeling TechniquesModeling Techniques Curve FittingCurve Fitting

• Easy to generate the modelEasy to generate the model• Very few exposed parametersVery few exposed parameters

ParameterizationParameterization• Requires one-time manual analysisRequires one-time manual analysis• Exposes many parametersExposes many parameters• Explore different system scenariosExplore different system scenarios

Coupling Coupling • Builds upon previous techniquesBuilds upon previous techniques• Identify how to combine kernel modelsIdentify how to combine kernel models


Kernel CouplingKernel Coupling

Two kernels (i & j)Two kernels (i & j) Three measurementsThree measurements

• PPii: performance of kernel i isolated: performance of kernel i isolated

• PPjj: performance of kernel j isolated: performance of kernel j isolated

• PPijij: performance of kernels i & j coupled: performance of kernels i & j coupled

Compute CCompute Cijij = = PjPi

Pij


Coupling CategoriesCoupling Categories

CCijij = 1: no coupling = 1: no coupling

CCijij > 1: destructive coupling > 1: destructive coupling

CCijij < 1: constructive coupling < 1: constructive coupling


Coupling CategoriesCoupling Categories

SharedSharedResourceResource

CCijij = 1: No Coupling = 1: No Coupling

CCijij < 1: Constructive Coupling < 1: Constructive CouplingCCijij > 1: Destructive Coupling > 1: Destructive Coupling

Kernel AKernel A Kernel BKernel B





Kernel AKernel A

Kernel BKernel B


Using Coupling ParametersUsing Coupling Parameters Use weighted averages to determine how to Use weighted averages to determine how to

combine coupling values combine coupling values Example:Example:

• Given the pair-wise coupling valuesGiven the pair-wise coupling values

Kernel AKernel A

Kernel BKernel B

Kernel CKernel C

Want:Want: T = E T = EAA + E + EBB + E + ECC1 2 3 = (C= (CABAB * P * PABAB + C + CACAC * P * PAC AC ))

PPABAB + P + PACAC

1

= (C= (CABAB * P * PABAB + C + CBCBC * P * PBC BC ))

PPABAB + P + PBCBC

2

= (C= (CBCBC * P * PBCBC+ C+ CACAC * P * PAC AC ))

PPBCBC + P + PACAC

3


Composition MethodComposition Method Synthetic kernels (array updates)Synthetic kernels (array updates)

Kernel A (196.44)Kernel A (196.44)

Kernel B (207.16)Kernel B (207.16)

Kernel C (574.19)Kernel C (574.19)

Kernel PairKernel Pair CouplingCoupling

A - BA - B 0.970.97

B - CB - C 0.750.75

C - AC - A 0.760.76

1 = 0.8472= 0.8472 2 = 0.8407= 0.8407 3 = 0.7591= 0.7591

Actual total timeActual total time: 799.63s: 799.63sCoupling timeCoupling time: 776.52s (Error: 2.89%): 776.52s (Error: 2.89%)Adding individual times: 971.81s (Error: 23%)Adding individual times: 971.81s (Error: 23%)


Coupling Method: UsageCoupling Method: UsageCoupling Method: UsageCoupling Method: Usage

Run


Inputs

Systems

Functions

Control Flow

Adjacent KernelsAdjacent Kernels

Coupling Coupling Values and Values and

Performance Performance datadata

Coupling

Data and Data and System InfoSystem Info

Analytical EquationAnalytical Equation(Octave: Coupling)(Octave: Coupling)


Case StudiesCase Studies

Predication: Resource AllocationPredication: Resource Allocation• Grid Physics Network (GriPhyN)Grid Physics Network (GriPhyN)• Utilizes Grid 2003 infrastructureUtilizes Grid 2003 infrastructure• GeoLIGO applicationGeoLIGO application

Prediction: Resource AllocationPrediction: Resource Allocation• AADMLSS: Educational ApplicationAADMLSS: Educational Application• Utilizes multiple serversUtilizes multiple servers


Case 1: GEO LIGOCase 1: GEO LIGO (GriPhyN) (GriPhyN)

The pulsar search is a process of finding celestial objects that may emit gravitational waves

• GEO (German-English Observatory) LIGO (Laser Interferometer Gravitational-wave Observatory) pulsar search is the most frequent coherent search method that generates F-statistic for known pulsars


GriPhyN GriPhyN Chimera

Virtual Data System

Prophesy

Grid Middleware Ganglia

GRID 2003

ResourceSelection

Monitoring

Transform using VDL

Submission


Resource Selector Resource Selector

Application Name

Input Parameters,List of available sites

Rankings of sites

Weights of each site

Prophesy

Interface

Predictor


Grid2003 TestbedGrid2003 Testbed


Execution EnvironmentExecution EnvironmentSite NameSite Name CPUsCPUs BatchBatch Compute NodesCompute Nodes

ProcessorsProcessors Cache Cache SizeSize

MemoryMemory

alliance.unm.edu (UNM)alliance.unm.edu (UNM) 436436 PBSPBS 1 X PIII 731 GHz1 X PIII 731 GHz 256 KB256 KB 1 GB1 GB

atlas.iu.edu (IU)atlas.iu.edu (IU) 400400 PBSPBS 2 X Intel Xeon 2.4 GHz2 X Intel Xeon 2.4 GHz 512 KB512 KB 2.5 GB2.5 GB

pdsfgrid3.nersc.gov (PDSF)pdsfgrid3.nersc.gov (PDSF) 349349 LSFLSF 2 X PIII 650-1.8 GHz 2 X PIII 650-1.8 GHz 2 X AMD 2100+ - 2600+2 X AMD 2100+ - 2600+

256 KB256 KB 2 GB2 GB

atlas.dpcc.uta.edu (UTA)atlas.dpcc.uta.edu (UTA) 158158 PBSPBS 2 X Intel Xeon 2.4 – 2.6 GHz 2 X Intel Xeon 2.4 – 2.6 GHz 512 KB512 KB 2 GB2 GB

nest.phys.uwm.edu (UWM)nest.phys.uwm.edu (UWM) 296296 CONDORCONDOR 1 X PIII 1GHz1 X PIII 1GHz 256 KB256 KB 0.5 GB0.5 GB

boomer1.oscer.ou.edu (OU)boomer1.oscer.ou.edu (OU) 286286 PBSPBS 3 X Intel Xeon 2 GHz3 X Intel Xeon 2 GHz 512 KB512 KB 2 GB2 GB

cmsgrid.hep.wisc.edu cmsgrid.hep.wisc.edu (UWMadison)(UWMadison)

6464 CONDORCONDOR 1 X Intel Xeon 2.8 GHz1 X Intel Xeon 2.8 GHz 512 KB512 KB 2 GB2 GB

cluster28.knu.ac.kr (KNU)cluster28.knu.ac.kr (KNU) 104104 CONDORCONDOR 1 X AMD Athlon XP 1700+1 X AMD Athlon XP 1700+ 256 KB256 KB 0.8 GB0.8 GB

acdc.ccr.buffalo.edu acdc.ccr.buffalo.edu (Ubuffalo)(Ubuffalo)

7474 PBSPBS 1 X Intel Xeon 1.6 GHz1 X Intel Xeon 1.6 GHz 256 KB256 KB 3.7 GB3.7 GB

ParametersParameters Prediction-basedPrediction-based Load-basedLoad-based RandomRandom

AlphaAlpha FreqFreq SiteSiteTimeTime(sec)(sec) SiteSite Time (sec)Time (sec) ErrorError Selected SiteSelected Site

Time Time (sec)(sec) ErrorError

0.00650.0065 0.0020.002 PDSFPDSF 3863.66 3863.66 UWMadisonUWMadison 9435.80 9435.80 59.05%59.05% UWMilwaukeeUWMilwaukee 48065.83 48065.83 60.09%60.09%

0.00850.0085 0.0010.001 IUIU 2850.39 2850.39 UWMadisonUWMadison 11360.28 11360.28 74.91%74.91% KNUKNU 7676.56 7676.56 62.87%62.87%

0.00750.0075 0.0090.009 IUIU 22090.17 22090.17 PDSFPDSF 20197.88 20197.88 -9.37%-9.37% UNMUNM 77298.13 77298.13 71.42%71.42%

0.00550.0055 0.0090.009 IUIU 16216.25 16216.25 UTAUTA 27412.45 27412.45 40.84%40.84% UWMadisonUWMadison 31555.10 31555.10 48.61%48.61%

0.00050.0005 0.0090.009 PDSFPDSF 1365.51 1365.51 UbuffaloUbuffalo 3226.00 3226.00 57.67%57.67% UWMilwaukeeUWMilwaukee 16009.82 16009.82 91.47%91.47%

0.00750.0075 0.0030.003 PDSFPDSF 6723.30 6723.30 IUIU 7343.37 7343.37 8.44%8.44% KNUKNU 8287.77 8287.77 18.88%18.88%

0.00650.0065 0.0070.007 PDSFPDSF 13561.01 13561.01 PDSFPDSF 13561.01 13561.01 0.00%0.00% UNMUNM 52379.31 52379.31 74.65%74.65%

0.00850.0085 0.0040.004 PDSFPDSF 10121.27 10121.27 UbuffaloUbuffalo 19649.22 19649.22 48.49%48.49% IUIU 11158.72 11158.72 9.30%9.30%

0.00350.0035 0.0050.005 PDSFPDSF 5241.28 5241.28 UbuffaloUbuffalo 20799.05 20799.05 74.80%74.80% UWMUWM 51936.49 51936.49 89.91%89.91%

0.00650.0065 0.0090.009 IUIU 19184.36 19184.36 UWMadisonUWMadison 24995.94 24995.94 23.25%23.25% OUOU 23441.16 23441.16 18.16%18.16%

0.00450.0045 0.0090.009IUIU 13278.68 13278.68 UTAUTA 20453.30 20453.30 35.08%35.08% UWMadisonUWMadison 14137.44 14137.44 6.07%6.07%

0.00850.0085 0.0090.009 IUIU 25021.39 25021.39 UWMadisonUWMadison 26246.68 26246.68 4.67%4.67% OUOU 31538.22 31538.22 20.66%20.66%

AverageAverage 　　　　 33.68%33.68% 　　 58.62%58.62%

Experimental ResultsExperimental Results


Case Study 2: AADMLSSCase Study 2: AADMLSS

African American Distributed Learning System (AADMLSS) developed by Dr. Juan E. GilbertAfrican American Distributed Learning System (AADMLSS) developed by Dr. Juan E. Gilbert


Site Selection Process Site Selection Process

Measure NetworkPerformance

Measure Server Performance

Select server with bestoverall site performance

User logs out

NO

NO

YES

Pass Quiz?

Next concept(same instructor)

Current concept(different instructor)

Exit?

YES

Display Concept

User logsinto AADMLSS

Valid Usernameand Password?

NO

YES

NO YES

First timeaccess?

Get default conceptGet last concept


Testbed OverviewTestbed Overview

CATEGORYCATEGORY SPECS SPECS Loner (TX)Loner (TX) Prophesy (TX)Prophesy (TX) Tina (MA)Tina (MA) Interact (AL)Interact (AL)

HardwareHardware

CPU Speed CPU Speed (MHz)(MHz)

997.62997.62 3056.853056.85 1993.561993.56 697.87697.87

Bus Speed Bus Speed (MB/s)(MB/s)

205205 856856 638638 214214

Memory (MB)Memory (MB) 256256 20482048 256256 256256

Hard Disk (GB)Hard Disk (GB) 3030 146146 4040 1010

SoftwareSoftware

O/SO/S Redhat Linux Redhat Linux 9.09.0

Redhat Linux Enterprise Redhat Linux Enterprise 3.03.0

Redhat Linux Redhat Linux 9.09.0

Redhat Linux Redhat Linux 9.09.0

Web ServerWeb Server Apache 2.0Apache 2.0 Apache 2.0Apache 2.0 Apache 2.0Apache 2.0 Apache 2.0Apache 2.0

Web Web ApplicationApplication

PHP 4.2PHP 4.2 PHP 4.3PHP 4.3 PHP 4.2PHP 4.2 PHP 4.1PHP 4.1


Course/Module/Course/Module/ConceptConcept

DAYDAY

NIGHTNIGHT

SRT-LOAD SRT-LOAD (%)(%)

SRT-SRT-RANDOM RANDOM

(%)(%)SRT-LOAD SRT-LOAD

(%)(%)SRT-RANDOM SRT-RANDOM

(%)(%)

3/0/03/0/0 9.759.75 16.9716.97 8.768.76 13.5413.54

3/0/13/0/1 12.5812.58 24.7624.76 12.3012.30 22.5422.54

3/0/23/0/2 16.7516.75 29.7029.70 15.7515.75 28.9528.95

3/0/33/0/3 20.5420.54 27.1027.10 18.7518.75 25.5425.54

3/1/03/1/0 9.149.14 16.9216.92 8.768.76 13.9613.96

3/1/13/1/1 8.678.67 15.7615.76 8.018.01 14.1514.15

3/1/23/1/2 13.3813.38 23.5723.57 11.9411.94 20.6720.67

3/1/33/1/3 12.1612.16 19.7619.76 11.8711.87 19.1119.11

3/2/03/2/0 8.958.95 15.1515.15 8.648.64 15.0915.09

3/2/13/2/1 11.5711.57 17.4017.40 9.959.95 15.5415.54

3/2/23/2/2 10.9510.95 19.7519.75 9.609.60 15.2715.27

3/2/33/2/3 11.0411.04 23.0823.08 12.5412.54 22.8422.84

3/3/03/3/0 8.918.91 15.9415.94 7.697.69 15.9115.91

3/3/13/3/1 9.079.07 17.9017.90 8.478.47 16.9516.95

3/3/23/3/2 9.469.46 16.7716.77 9.319.31 15.7615.76

3/3/33/3/3 10.5510.55 19.5719.57 9.879.87 17.9517.95

AVERAGEAVERAGE 11.4711.47 20.0120.01 10.7610.76 18.3618.36

4-Servers4-Servers


Results - 4 ServersResults - 4 Servers

0%

25%

50%

75%

100%

Random (D) Random (N) Load (D) Load (N) SRT (D) SRT (N)

Site Selection Distribution

Loner

Prophesy

Tina

Interact


Results – 3 Servers Results – 3 Servers ConceptConcept SRT-LOAD (%)SRT-LOAD (%) SRT-RANDOM (%)SRT-RANDOM (%)

3/0/0 D3/0/0 D 6.216.21 14.0514.05

3/0/1 D3/0/1 D 12.1312.13 21.9421.94

3/0/2 N3/0/2 N 14.0214.02 25.8325.83

3/0/3 N3/0/3 N 18.1218.12 23.5223.52

3/1/0 N3/1/0 N 8.058.05 12.0412.04

3/1/1 N3/1/1 N 7.317.31 12.2512.25

3/1/2 N3/1/2 N 12.6012.60 18.7418.74

3/1/3 N3/1/3 N 10.9610.96 19.1119.11

3/2/0 N3/2/0 N 7.937.93 12.5812.58

3/2/1 N3/2/1 N 8.058.05 14.2514.25

3/2/2 N3/2/2 N 9.149.14 15.9715.97

3/2/3 D3/2/3 D 9.799.79 20.5820.58

3/3/0 D3/3/0 D 8.948.94 13.6413.64

3/3/1 D3/3/1 D 8.268.26 16.7416.74

3/3/2 D3/3/2 D 9.219.21 15.2115.21

3/3/3 D3/3/3 D 9.979.97 19.3619.36

AVERAGEAVERAGE 10.0410.04 17.2417.24

0%

25%

50%

75%

100%

Random (D) Random(N) Load (D) Load (N) SRT (D) SRT (N)

Site Selection Distribution

Loner

Tina

Interact


Results – 3 ServersResults – 3 Servers

0

500

1000

1500

2000

2500

3000

msec

Lo

ner

Tin

aIn

tera

ct

Lo

ner

Tin

aIn

tera

ct

Lo

ner

Tin

aIn

tera

ct

Lo

ner

Tin

aIn

tera

ct

Lo

ner

Tin

aIn

tera

ct

Lo

ner

Tin

aIn

tera

ct

Lo

ner

Tin

aIn

tera

ct

Lo

ner

Tin

aIn

tera

ct

3/0/0 D 3/0/1 D 3/0/2 N 3/0/3 N 3/2/0 N 3/2/1 N 3/2/2 D 3/2/3D

Average Service Response Time - AGENT

Netw ork Delay

Server Access Time


Results – 2 Servers Results – 2 Servers

0%

25%

50%

75%

100%

Random Load SRT

Site Selection Distribution (DAY)

Tina

Interact

ConceptConcept SRT-LOAD (%)SRT-LOAD (%) SRT-RANDOM (%)SRT-RANDOM (%)

3/0/0 D3/0/0 D 3.133.13 4.034.03

3/0/1 D3/0/1 D 4.264.26 5.975.97

3/0/2 D3/0/2 D 7.027.02 8.288.28

3/0/3 D3/0/3 D 8.648.64 9.029.02

3/1/0 D3/1/0 D 3.253.25 4.944.94

3/1/1 D3/1/1 D 3.273.27 4.104.10

3/1/2 D3/1/2 D 3.933.93 5.975.97

3/1/3 D3/1/3 D 3.643.64 4.084.08

3/2/0 D3/2/0 D 3.153.15 3.323.32

3/2/1 D3/2/1 D 4.394.39 5.205.20

3/2/2 D3/2/2 D 5.805.80 5.975.97

3/2/3 D3/2/3 D 6.526.52 6.956.95

3/3/0 D3/3/0 D 4.394.39 5.645.64

3/3/1 D3/3/1 D 4.164.16 5.205.20

3/3/2 D3/3/2 D 4.814.81 5.735.73

3/3/3 D3/3/3 D 5.025.02 5.585.58

AVERAGEAVERAGE 4.714.71 5.625.62


SummarySummary ProphesyProphesy Two case studies with resource allocationTwo case studies with resource allocation

• Geo LIGO: on average 33% better than load-Geo LIGO: on average 33% better than load-based selectionbased selection

• AADMLSS: on average 4-11% better than load-AADMLSS: on average 4-11% better than load-based selectionbased selection

Future workFuture work• Continue extending application baseContinue extending application base• Work on queue wait time predictionsWork on queue wait time predictions


Performance Analysis ProjectsPerformance Analysis Projects ProphesyProphesy

• http://prophesy.cs.tamu.eduhttp://prophesy.cs.tamu.edu• Published over 20 conference and journal papers Published over 20 conference and journal papers

PAPIPAPI• http://icl.cs.utk.edu/papi/http://icl.cs.utk.edu/papi/

SCALEA-GSCALEA-G• http://www.dps.uibk.ac.at/projects/scaleag/http://www.dps.uibk.ac.at/projects/scaleag/

PerfTrackPerfTrack• http://web.cecs.pdx.edu/~karavan/perftrackhttp://web.cecs.pdx.edu/~karavan/perftrack

ParadynParadyn• http://www.cs.wisc.edu/~paradyn/http://www.cs.wisc.edu/~paradyn/

Network Weather ServiceNetwork Weather Service• http://nws.cs.ucsb.eduhttp://nws.cs.ucsb.edu

prophesy: analysis and modeling of parallel and distributed applications

Documents

performance of kernels

destructive couplingcij

constructive couplingcij

distributed systems

modeling of parallel

kernel modelshttp

system databasescurrently

constructive couplinghttp