apples, nws and the ipg fran berman ucsd and npaci rich wolski ucsd, u. tenn. and npaci this...
Post on 20-Dec-2015
213 views
TRANSCRIPT
AppLeS, NWS and the IPG
Fran Berman
UCSD and NPACI
Rich Wolski
UCSD, U. Tenn. and NPACI
AppLeS and the IPG
Usability,Integration
development ofbasic IPG infrastructure
Development of persistent IPG testbed
Performance
“IPG - aware”programming
Short-term Medium-term Long-term
Application schedulingResource schedulingThroughput scheduling
Multi-schedulingResource economy
Integration of schedulers and other tools, performanceinterfaces
Experience withPilot IPG
Development of prototype performance-oriented applications
Development of necessary research
A Model for the Future• Adaptation is key to the ultimate IPG program
development and execution environment.
• Exchange of performance information fundamental to the success of IPG applications
PSE
Config.object
program
wholeprogramcompiler
Source appli-cation
libraries
Realtimeperf
monitor
Dynamicoptimizer
Grid runtime system
negotiation
Softwarecomponents
Service negotiator
Scheduler
Performance feedback
Perfproblem
Grid Application Development System (GrADS)
Why Application Schedulers?
• Application performance can conflict with performance goals of other system components
• Goal of application scheduler is to prioritize performance of the application over other system components
Agent-based Application Scheduling
Sensor Interface
Reporting Interface
Forecaster
Model ModelModel
NWSUserPrefs
AppPerf
Model
PlannerResource Selector
Application
Act.
IPG /Globus infrastructure
NWS (Wolski)AppLeS (Berman and Wolski)
Performance Prediction
• Given monitored bandwidth data, what will happen next?
Fast Ethernet Bandwidth at SDSC
0
10
20
30
40
50
60
70
Time of Day
Meg
abits
per
Sec
ond
Measurements
Tue Wed Thu Fri Sat Sun Mon Tue
13:30
NWS Predictions• Monitored data provides a snapshot of what has
happened.
• What we really want to know is: What will happen?
Fast Ethernet Bandwidth at SDSC
0
10
20
30
40
50
60
70
Time of Day
Me
ga
bits
pe
r S
eco
nd
Measurements
Exponential SmoothingPredictions
Tue Wed Thu Fri Sat Sun Mon Tue
Monitoring vs. Prediction
Mean Square Error PerformanceSDSC Ethernet
0
20
40
60
80
100
120
140MSE
• Last value not always the best predictor• Hard to develop accurate forecasting models -- why
not use all feasible models?
Monitored data
Do AppLeS and NWS Improve Application Performance?
• Good results with many applications including
– SARA AppLeS
– CompLib AppLeS
– Jacobi2D AppLeS
• AppLeS/NWS applications demonstrate that
– prediction is possible in high-variance environments
– adaptivity can improve performance
SARA AppLeS
• SARA = Synthetic Apperture Radar Atlas– application developed at
JPL and SDSC
• Goal: Process radar images from distributed database for user’s desired image
• AppLeS focuses on resource selection problem
. . .
ComputeServers
DataServers
Client
SARA Experiments
CompLib AppLeS
• Problem: Find the best matches between two gene sequence libraries
• Apply FASTA algorithm to all sequence pairs to determine similarity
• Developed for DOCT testbed
sequence library
sequ
ence
libr
ary
Execution time
0
50
100
150
200
250
300
350
Small Medium Large
Problem Size
Tim
e (s
)
SuperAppLeSAppLeSMentat
CompLib Experiments
Jacobi2D AppLeS• Important component of
many scientific applications
• Time-balancing used to achieve minimal execution time
• Scheduler solves time-balancing equations for Area
iii Commpt
OperAreaT
N N Areai
Jacobi2D Experiments• Comparison of AppLeS with and without NWS
info, and load-balancing
0
1
2
3
4
5
6
7
Exe
cuti
on T
ime
(sec
onds
)
1000
1100
1200
1300
1400
1500
1600
1700
1800
1900
2000
Problem Size
Comparison of Execution Times
Compile-time Blocked
Compile-time Irregular Strip
Runtime
Applying AppLeS/NWS Methodology to the IPG
• AppLeS/NWS methodology can be used to develop performance-efficient IPG applications
• IPG FY99 projects leverage FY98 project and previous AppLeS/NWS development and research
IPG FY99 Project: A “Parameter Sweep” Template
• INS2D representative of larger class of critical NASA applications
• AppLeS parameter sweep template will build on INS2D model and experiments to target larger class of applications and platforms
• Template will serve as a prototype IPG PSE workbench tool
AppLe S
AP
I
Resources
App-specific
case
gen.
Exp
Act
ActSched.
Act
Exp Exp
AppLeS Project Plan FY99 (Berman,UCSD)
• Expand INS2D AppLeS– to NASA IPG testbed
– to include batch systems
– to target Globus
• Development of Parameter Sweep AppLeS template• Goal: To provide framework for improving turnaround time of parameter study
component of complex AES applications
• AppLeS scheduling agents prototype autonomous agent technology for IPG
• Requires development of strategy for scheduling in mixed batch and interactive environments
Project Personnel: Berman, Casanova (UCSD)Collaborators: Wolski (U. Tenn.), Kesselman (ISI/USC)
NWS Project Plan FY99 (Wolski, U. Tenn.)
• Enhance the NWS to support AppLeS parameter sweep template in NASA Globus environment
– NWS API for parameter sweep template
– integration with Globus
• Integrate NWS with IPG and Globus application performance monitoring tools
– use NWS performance techniques to predict application performance dynamically
• Investigate strategies for monitoring and forecasting batch system performance
– queue wait times in the presence of user priorities, etc.
Project Personnel: Wolski (U. Tenn)Collaborators: Berman (UCSD), Moore (SDSC), Kesselman (ISI/USC)
Possible Additional IPG Projects
• AppLeS/NWS-enhanced Storage Resource Broker
Project: Enhance SRB performance through agent-based
scheduling
Project Personnel: Berman, Wolski
Collaborator: Moore
• AppLeS/NWS-enhanced NetSolve over Globus
Project: Improve scheduling component of NetSolve using
AppLeS/NWS techniques, deploy on Globus IPG platform
Project Personnel: Berman, Wolski, Casanova, Dongarra
Collaborator: Kesselman
Possible Additional IPG Projects
• AppLeS/NWS Applications on Condor
Project: Develop AppLeS application which can achieve
performance in the Condor environment; integrate
Condor and NWS information; leverage Condor/Globus
integration
Project Personnel: Berman, Wolski
Collaborator: Livny, Kesselman
Project Information• NWS Home Page:
http://nws.npaci.edu
• AppLeS + NWS Project Personnel
– Francine Berman– Rich Wolski– Walfredo Cirne– Marcio Faerman– Jaime Frey– Jim Hayes– Graziano Obertelli
• AppLeS Home Page: http://www-cse.ucsd.edu/groups/hpcl/apples.html
– Jenny Schopf– Gary Shao– Neil Spring – Shava Smallen– Alan Su– Dmitrii Zagorodnov