performance architecture within iceni dr andrew stephen m c gough laurie young, ali afzal, steven...
TRANSCRIPT
![Page 1: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/1.jpg)
Performance Architecture within ICENI
Dr Andrew Stephen McGoughLaurie Young, Ali Afzal, Steven Newhouse and John Darlington
London e-Science Centre
Department of Computing, Imperial College London
![Page 2: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/2.jpg)
2
Outline
• Overview of ICENI
• Performance Framework
• Example
• Conclusion
![Page 3: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/3.jpg)
3
ICENI: Imperial College e-Science Network Infrastructure
• Collect and provide relevant Grid Meta-Data
• Pluggable architecture• Test Architecture for Grid Research• Foundation for higher-level Services
and Autonomous Composition• Integrated Grid Middleware Solution• Interoperability between
architectures, APIs • Added value layer to other
middleware• Usability: Interactive Grid Workflows• Role and policy driven security• ICENI Open Source licence (extended
SISSL)
The Iceni, under Queen Boudicca, united the tribes of South-East England in a revolt against the occupying Roman forces in AD60.
![Page 4: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/4.jpg)
4
Scheduling Workflows in ICENI
• Applications consist of a number of components linked together in a dataflow manner
• The abstract workflow needs to be mapped down to a set of component implementations which will run on resources
Linear EquationSource
Linear EquationSolver
Display VectorResults
General Equation generato
r
LU Factorisatio
n
Simple Vector Display
![Page 5: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/5.jpg)
5
The Problem
• We have:– Multiple resources where components can run– Multiple implementations of components– The choice of one resource component
mapping can affect the others
• User wants predictable performance
• How to choose the “best” mapping of workflow over resources to give user predictability?
![Page 6: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/6.jpg)
6
The Solution
• We need to take into account:– Execution Times of components on Resources
• Performance Data
– Inter-component effects of workflows• Workflow aware Schedulers
– Workload on resources, making sure they are free when we need them
• Reservation systems
![Page 7: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/7.jpg)
7
The Architecture
Scheduler
ReservationService
PerformanceStore
Launcher
ApplicationService
ReservationEngine
Workflow
![Page 8: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/8.jpg)
8
Performance Store- Persistent Performance storage
Performance Store- Persistent Performance storage
The Performance Repository Framework:
Performance Framework
Performance Processing- Conversion of raw event times into performance data
Data Collector-Collecting data on currently running applications (event times)
Performance Store- Persistent Performance storage
![Page 9: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/9.jpg)
9
Collection of Performance Results
Data CollectorLinear EquationSource
Linear EquationSolver
Display VectorResults
Time Event 12:00 Linear Equation Source Start 12:04 Send out Equations 12:03 Linear Equation Solver Start 12:05 Receive Equations 12:12 ………..
Event:Start LinearEquation Source
Performance Processing
Workflow
Performance Store
![Page 10: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/10.jpg)
10
Storing Performance Data
• Multiple stores can be used in one framework
• The stores may be data stores or analytical models
• All assumed to be persistent• Allows requests for predictions to be made• New Data can be added to the stores• Store data is aggregated together
– based upon reliability of store data• Provided by the store
Performance Store
![Page 11: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/11.jpg)
11
Using Performance Data
• Scheduler Builds up workflow graph with timings requested from the Performance Repository
• Timings are based on component implementation, resource, co-allocation count and other properties defined by the component implementer
• As the store will not contain all possible combinations of these properties regression is used to provide estimates for the missing values– This is an area of ongoing research
![Page 12: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/12.jpg)
12
Reservation Engine
ReservationEngine Framework
Reservations Database- Data about upcoming reservations,usedto test if a new reservation can be made
DRM Translation level-Communication with the underlying DRM system for supporting Reservations
Listen out for requests- Launcher services wishing to makereservations
![Page 13: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/13.jpg)
13
Reservation Service
ReservationService Framework
Reservations Database- Data about upcoming reservations,usedto test if a new reservation can be made
Reservation orchestration-Attempts to reserve all resources for a given workflow. May shift when it runs
Listen out for Services- Launcher with reservation- Scheduling Services
![Page 14: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/14.jpg)
14
Reservations
time →
time →
ReservationService
Scheduler
Workflow Reserve (workflow)
HOLD
Conflict
HOLD
HOLD
Reserve
Reserve
WS-AgreementRequest interval
Linear EquationSource
Linear EquationSolver
Display VectorResults
Reservations not possible on Users Desktop
![Page 15: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/15.jpg)
15
Example: Linear Equation Solver
service list composition pane parameters
![Page 16: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/16.jpg)
16
Inferring Workflow from Dataflow
![Page 17: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/17.jpg)
17
Conclusion
• Better usage of resources.– Reservations of resources in the future– Determining if co-allocation of components will affect
performance– Late Enactment of components
• Critical Path analysis – can schedule this appropriately
• Provides a framework for experimentation with– Different scheduling algorithms– Different performance models– Different reservation policies
![Page 18: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/18.jpg)
18
Performance Trinity
Performance Models
Grid Scheduling Reservation Fabric
![Page 19: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/19.jpg)
19
The Architecture: Showing the Trinity
SchedulerReservation
Service
PerformanceStore
Launcher
ApplicationService
ReservationEngine
![Page 20: Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre](https://reader030.vdocument.in/reader030/viewer/2022032805/56649ee85503460f94bf9993/html5/thumbnails/20.jpg)
20
Acknowledgements
• Director: Professor John Darlington• Research Staff:
– Nathalie Furmento, Stephen McGough, William Lee– Jeremy Cohen, Marko Krznaric, Murtaza Gulamali– Asif Saleem, Laurie Young, Jeffrey Hau– David McBride, Ali Afzal
• Support Staff:– Oliver Jevons, Sue Brookes, Glynn Cunin, Keith Sephton
• Alumni:– Steven Newhouse, Yong Xie, Gary Kong– James Stanton, Anthony Mayer, Angela O’Brien
• Contact:– http://www.lesc.ic.ac.uk/ e-mail: [email protected]