economics paradigm for resource management and scheduling for service oriented p2p/grid computing...
TRANSCRIPT
![Page 1: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/1.jpg)
Economics Paradigm for “Resource Management and
Scheduling” for Service Oriented P2P/Grid
Computing
Rajkumar Buyya
Melbourne, Australiawww.buyya.com/ecogrid
WW Grid
![Page 2: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/2.jpg)
2
![Page 3: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/3.jpg)
3
Need Honest Answers!
I want to have access to your Grid resources & want to knowhow many of you are willing to give me access ? (following cases)
I am unable to give you access our Australian machines, but I want to have access to yours! [social]
Want to solve academic problems Want to solve business problems
I am willing to gift you Kangaroos! [bartering] I am willing to give you access to my machines, if you
want. (sharing, but no measure & no QoS) [bartering] I am willing to pay you dollars on usage basis.
[economic incentive, market-based, and QoS]
WW Grid
![Page 4: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/4.jpg)
4
Overview
A quick glance at today’s Grid computing Resource Management challenges for next
generation Grid computing A Glance at Approaches to Grid computing. Grid Architecture for Computational Economy Economy Grid = Globus + GRACE Nimrod-G -- Grid Resource Broker Scheduling Experiments Case Study: Drug Design
Application on Grid Conclusions
Scheduling Economics
Grid
EconomyGrid
![Page 5: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/5.jpg)
5
2100
2100 2100 2100 2100
2100 2100 2100 2100
Desktop SMPs or SuperComputers
LocalCluster
GlobalCluster/Grid
PERFORMANCE
Inter PlanetaryGrid!
•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Universe
Administrative Barriers
EnterpriseCluster/Grid
?
Scalable HPC: Breaking Administrative Barriers & new challenges
![Page 6: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/6.jpg)
6
Why Grids? Large Scale Explorations need them—Killer
Applications. Solving grand challenge applications using
modeling, simulation and analysis
Life Sciences
CAD/CAM
Aerospace
Military ApplicationsDigital Biology Military ApplicationsMilitary Applications
Internet & Ecommerce
![Page 7: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/7.jpg)
7
![Page 8: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/8.jpg)
8
What is Grid ?
An infrastructure that logically couples distributed resources:
Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc;
Software – e.g., ASPs renting expensive special purpose applications on demand;
Catalogued data and databases – e.g. transparent access to human genome database;
Special devices – e.g., radio telescope – SETI@Home searching for life in galaxy.
People/collaborators. and presents them as an integrated global resource. It enables the creation of virtual enterprises (VEs)
for resource sharing.
Widearea
data archives
![Page 9: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/9.jpg)
9
P2P/Grid Applications-Drivers
Distributed HPC (Supercomputing): Computational science.
High-Capacity/Throughput Computing: Large scale simulation/chip design & parameter studies.
Content Sharing (free or paid) Sharing digital contents among peers (e.g., Napster)
Remote software access/renting services: Application service provides (ASPs) & Web services.
Data-intensive computing: Virtual Drug Design, Particle Physics, Stock Prediction...
On-demand, realtime computing: Medical instrumentation & Mission Critical.
Collaborative Computing: Collaborative design, Data exploration, education.
Service Oriented Computing (SOC): Computing as Utility: New paradigm and new industries.
![Page 10: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/10.jpg)
10
Building and Using Grids require
Services that make our systems Grid Ready! Security mechanisms that permit resources
to be accessed only by authorized users. (New) programming tools that make our
applications Grid Ready!. Tools that can translate the requirements of
an application/user into the requirements of computers, networks, and storage.
Tools that perform resource discovery, trading, selection/allocation, scheduling and distribution of jobs and collects results.
Globus
?
![Page 11: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/11.jpg)
11
Players in Grid Computing
![Page 12: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/12.jpg)
12
What users want ?Users in Grid Economy &
Strategy Grid Consumers
Execute jobs for solving varying problem size and complexity
Benefit by selecting and aggregating resources wisely Tradeoff timeframe and cost
Strategy: minimise expenses Grid Providers
Contribute “idle” resource for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity
Strategy: maximise return on investment
![Page 13: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/13.jpg)
Challenges for Next Generation Grid
Technology Development
![Page 14: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/14.jpg)
14
Challenges for Grid Computing
Security
Resource Allocation & Scheduling
Data locality
Network Management
System Management
Resource Discovery
Uniform Access
Computational Economy
Application Development Tools
![Page 15: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/15.jpg)
15
Sources of Complexity in Resource Management for World Wide Grid
Computing Size (large number of nodes, providers, consumers) Heterogeneity of resources (PCs, Workstations, clusters, and
supercomputers, instruments, databases, software) Heterogeneity of fabric management systems (single system image OS,
queuing systems, etc.) Heterogeneity of fabric management polices Heterogeneity of application requirements (CPU, I/O, memory, and/or
network intensive) Heterogeneity in resource demand patterns (peak, off-peak, ...) Applications need different QoS at different times (time critical results). The
utility of experimental results varies from time to time. Geographical distribution of users & located different time zones Differing goals (producers and consumers have different objectives and
strategies) Unsecure and Unreliable environment
![Page 16: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/16.jpg)
16
Traditional approaches to resource management & scheduling are NOT useful
for Grid ? They use centralised policy that need
complete state-information and common fabric management policy or decentralised consensus-based
policy. Due to too many heterogenous parameters in the Grid it is impossible to
define/get: system-wide performance matrix and common fabric management policy that is acceptable to all.
“Economics” paradigm proved to effective institution in managing decentralization and heterogeneity that is present in human economies!
Fall of USSR & Emergence of US as world superpower! (monopoly?) So, we propose/advocate the use of computational economics principles
in management of resources and scheduling computations on world wide Grid.
Think locally and act globally approach to grid computing!
![Page 17: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/17.jpg)
17
Benefits of Computational Economies
It provides a nice paradigm for managing self interested and self-regulating entities (resource owners and consumers)
Helps in regulating supply-and-demand for resources. Services can be priced in such a way that equilibrium is maintained.
User-centric / Utility driven: Value for money! Scalable:
No need of central coordinator (during negotiation) Resources(sellers) and also Users(buyers) can make their own decisions and try to
maximize utility and profit. Adaptable It helps in offering different QoS (quality of services) to different applications
depending the value users place on them. It improves the utilisation of resources It offers incentive for resource owners for being part of the grid! It offers incentive for resource consumers for being good citizens There is large body of proven Economic principles and techniques available, we can
easily leverage it.
![Page 18: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/18.jpg)
18
New challenges of Computational Economy
Resource Owners How do I decide prices ? (economic models?) How do I specify them ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …..
Resource Consumers How do I decide expenses ? How do I express QoS requirements ? How I trade between timeframe & cost ? ….
Any tools, traders & brokers available to automate the process ?
![Page 19: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/19.jpg)
19
mix-and-match
Object-oriented
Internet/partial-P2P
Network enabled Solvers
Market/Computational Economy
![Page 20: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/20.jpg)
20
Many Grid Projects & Initiatives
Australia Economy Grid Nimrod-G Virtual Lab Active Sheets DISCWorld ..new coming up
Europe UNICORE MOL Lecce GRB Poland MC Broker EU Data Grid EuroGrid MetaMPI Dutch DAS XW, JaWS and many more...
Japan Ninf DataFarm and many more...
USA Globus Legion Javelin AppLeS NASA IPG Condor Harness NetSolve AccessGrid GrADS and many more...
Cycle Stealing & .com Initiatives Distributed.net SETI@Home, …. Entropia, UD, Parabon,….
Public Forums Global Grid Forum P2P Working Group IEEE TFCC Grid & CCGrid conferences
http://www.gridcomputing.com
![Page 21: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/21.jpg)
21
Many Testbeds ? & who pays ?,
who regulates supply and demand ?
GUSTO (decommissioned)
Legion Testbed
NASA IPG
World Wide Grid
WW Grid
![Page 22: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/22.jpg)
22
Testbeds so far -- observations
Who contributed resources & why ? Volunteers: for fun, challenge, fame, charismatic apps, public
good like distributed.net & SETI@Home projects. Collaborators: sharing resources while developing new
technologies of common interest – Globus, Legion, Ninf, Ninf, MC Broker, Lecce GRB,... Unless you know lab. leaders, it is impossible to get access!
How long ? Short term: excitement is lost, too much of admin. Overhead
(Globus inst+), no incentive, policy change,… What we need ? Grid Marketplace!
Regulates supply-and-demand, offers incentive for being players, simple, scalable solution, quasi-deterministic – proven model in real-world.
![Page 23: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/23.jpg)
23
Building an Economy Grid(Next Generation Grid
Computing!)
To enable the creation and promotion of:Grid Marketplace (competitive)
ASPService Oriented Computing
. . .And let users focus on their own work (science, engineering, or commerce)!
![Page 24: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/24.jpg)
24
Grid Node N
GRACE: A ReferenceGrid Architecture for Computational Economy
Grid User
Application
Grid Resource Broker
Grid Service Providers
Grid Explorer
Schedule Advisor
Trade Manager
Job ControlAgent
Deployment Agent
Trade Server
Resource Allocation
ResourceReservation
R1
Misc. services
Information Server(s)
R2 Rm…
Pricing Algorithms
Accounting
Grid Node1
…
Grid Middleware Services
…
…
HealthMonitor
Grid Market Services
JobExec
Info ?
Secure
Trading
QoS
Storage
Sign-on
Grid Bank
See PDPTA 2000 paper!
![Page 25: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/25.jpg)
25
Economic Models
Price-based: Supply,demand,value, wealth of economic system
Commodity Market Model Posted Price Model Bargaining Model Tendering (Contract Net) Model Auction Model
English, first-price sealed-bid, second-price sealed-bid (Vickrey), and Dutch (consumer:low,high,rate; producer:high, low, rate)
Proportional Resource Sharing Model Monopoly (one provider) and Oligopoly (few players)
consumers may not have any influence on prices. Bartering
Shareholder Model Partnership Model
See SPIE ITCom 2001 paper!: with Heinz Stockinger, CERN!
![Page 26: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/26.jpg)
26
Grid Open Trading Protocols
Get Connected
Call for Bid(DT)
Reply to Bid (DT)
Negotiate Deal(DT)
Confirm Deal(DT, Y/N)
….
Cancel Deal(DT)
Change Deal(DT)
Get Disconnected
Trade Manager Trade Server
Pricing Rules
DT - Deal Template - resource requirements (BM) - resource profile (BS) - price (any one can set) - status - change the above values - negotiation can continue - accept/decline - validity period
API
![Page 27: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/27.jpg)
27
GridFabric
GridApps.
GridMiddleware
GridTools
Networked Resources across Organisations
Computers Clusters Data Sources Scientific InstrumentsStorage Systems
Local Resource Managers
Operating Systems Queuing Systems TCP/IP & UDP
…
Libraries & App Kernels …
Distributed Resources Coupling Services
Security Information … QoSProcess
Development Environments and Tools
Languages Libraries Debuggers … Web toolsResource BrokersMonitoring
Applications and Portals
Prob. Solving Env.Scientific …CollaborationEngineering Web enabled Apps
Resource Trading
Grid Components
Market Info
![Page 28: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/28.jpg)
28
Economy Grid = Globus + GRACE
Applications
MDS
GRAMGlobus Security Interface
Heartbeat MonitorNexus
Local Services
LSF
Condor GRD QBank
PBS
TCP
SolarisIrixLinux
UDP
High-level Services and Tools
DUROC globusrunMPI-G Nimrod/GCC++
Grid Status
GASS
GRACE-TS
GARA
GridFabric
GridApps.
GridMiddleware
GridTools
GBankGMD
eCash
JVM
DUROC
Core Services
Science
Engineering Commerce Portals ActiveSheet……
See IPDPS HWC 2001 paper!
……
![Page 29: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/29.jpg)
29
GRACE components
A resource broker (e.g., Nimrod/G) Various resource trading protocols for different
economic models A mediator for negotiating between users and
grid service providers (Grid Market Directory) A deal template for specifying resource
requirements and services offers Grid Trading Server Pricing policy specification Accounting (e.g., QBank) and payment
management (GridBank, not yet implemented)
![Page 30: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/30.jpg)
30
Pricing, Accounting, Allocations and Job Scheduling Flow @ each site/Grid Level
QBankQBank
Resource Manager44
IBM-LL/PBS/….
00
55 88
66 77
Compute Resourcesclusters/SGI/SP/...
0. Make Deposits, Transfers, Refunds, Queries/Reports1. Clients negotiates for access cost.2. Negotiation is performed per owner defined policies. 3. If client is happy, TS informs QB about access deal.4. Job is Submitted5. Check with QB for “go ahead”6. Job Starts7. Job Completes8. Inform QB about resource resource utilization.
Trade Server 3311
Pricing PolicyPricing Policy22
DB@Each SiteDB@Each Site
GRID BankGRID Bank(digital transactions)(digital transactions)00
![Page 31: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/31.jpg)
31
Service Items to be Charged
CPU - User and System time Memory:
maximum resident set size - page size amount of memory used page faults: with/without physical I/O
Storage: size, r/w/block IO operations Network: msgs sent/received Signals received, context switches Software and Libraries accessed Data Sources (e.g. Protein Data Bank)
![Page 32: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/32.jpg)
32
How to decide Price ?
Fixed price model (like today’s Internet) Dynamic/Demand and Supply (like tomorrow’s Internet) Usage Period Loyalty of Customers (like Airlines favoring frequent flyers!) Historical data Advance Agreement (high discount for corporations) Usage Timing (peak, off-peak, lunch time) Calendar based (holiday/vacation period) Bulk Purchase (register 100 .com domains at once!) Voting -- trade unions decide pricing structure Resource capability as benchmarked in the market! Academic R&D/public-good application users can be offered at
cheaper rate compared to commercial use. Customer Type – Quality or price sensitive buyers. Can be Prescribed by Regulating (Govt.) authorities
![Page 33: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/33.jpg)
33
Payments- Options & Automation
Buy credits in advance / GSPs bill the user later--”pay as you go”
Pay by Electronic Currency via Grid Bank NetCash (anonymity), NetCheque, and Paypal NetCheque: - http://www.isi.edu/gost/info/netcash/
Users register with NC accounting servers, can write electronic cheques and send (e.g email). When deposited, balance is transferred from sender to receiver account.
NetCash - http://www.isi.edu/gost/info/netcheque/ It supports anonymity and it uses the NetCheque system to
clear payments between currency servers. Paypal.com– account+email is linked to credit card.
Enter the recipient’s email address and the amount you wish to request.
The recipient gets an email notification and pays you at www.PayPal.com
![Page 34: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/34.jpg)
Nimrod-G:The Grid Resource Broker
Soft Deadline and Budget-based Economy Grid Resource Broker
for Parameter Processing on P2P Grids
![Page 35: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/35.jpg)
35
Parametric Computing(What Users think of Nimrod
Power)
Multiple RunsSame ProgramMultiple Data Killer Application for the Grid!
ParametersAge Hair
23 CleanAge Hair
23 Clean23 Beard28 Goatee
Age Hair23 Clean23 Beard
Age Hair23 Clean23 Beard28 Goatee28 Clean
Age Hair23 Clean23 Beard28 Goatee28 Clean19 Moustache
Age Hair23 Clean23 Beard28 Goatee28 Clean19 Moustache10 Clean
Age Hair23 Clean23 Beard28 Goatee28 Clean19 Moustache10 Clean
-4000000 Too much
Courtesy: Anand Natrajan, University of Virginia
Magic Engine
See IPDPS 2000 paper!
![Page 36: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/36.jpg)
36
P-study Applications -- Characteristics
Code (Single Program: sequential or threaded)
High Resource Requirements Long-running Instances Numerous Instances (Multiple Data) High Computation-to-Communication
Ratio Embarrassingly/Pleasantly Parallel
![Page 37: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/37.jpg)
37
Sample P-Sweep ApplicationsSample P-Sweep Applications
Bioinformatics: Bioinformatics: Drug Design / Protein Drug Design / Protein
ModellingModelling
SensitivitySensitivityexperiments experiments
on smog formationon smog formation
Combinatorial Combinatorial Optimization:Optimization:
Meta-heuristic Meta-heuristic parameter estimationparameter estimation
Ecological Modelling: Ecological Modelling: Control Strategies Control Strategies
for Cattle Tickfor Cattle Tick
Electronic CAD: Electronic CAD: Field Programmable Field Programmable
Gate ArraysGate ArraysComputer Graphics: Computer Graphics: Ray TracingRay Tracing
High Energy High Energy Physics: Physics:
Searching for Searching for Rare EventsRare Events
Finance: Finance: Investment Risk AnalysisInvestment Risk Analysis
VLSI Design: VLSI Design: SPICE SimulationsSPICE Simulations
Aerospace: Aerospace: Wing DesignWing Design
Network SimulationNetwork SimulationAutomobile:Automobile:
Crash Simulation Crash Simulation
Data MiningData Mining
Civil Engineering:Civil Engineering:Building Design Building Design
astrophysics astrophysics
![Page 38: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/38.jpg)
38
Thesis
Perform parameter sweep (bag of tasks) (utilising distributed resources) within “T” hours or early and cost not exceeding $M.
Three Options/Solutions: Using pure Globus commands Build your own Distributed App & Scheduler Use Nimrod-G (Resource Broker)
![Page 39: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/39.jpg)
39
Remote Execution Steps
Choose Resource
Transfer Input Files
Set Environment
Start Process
Pass Arguments
Monitor Progress
Read/Write Intermediate Files
Transfer Output Files
Summary ViewJob ViewEvent View
+Resource Discovery, Trading, Scheduling, Predictions, Rescheduling, ...
![Page 40: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/40.jpg)
40
Using Pure Globus commands
Do all yourself! (manually)
Total Cost:$???
![Page 41: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/41.jpg)
41
Build Distributed Application & Scheduler
Build App case by case basisComplicated Construction
E.g., AppLeS/MPI based Total Cost:$???
![Page 42: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/42.jpg)
42
Use Nimrod-G
Aggregate Job SubmissionAggregate View
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
East
West
North
South
Submit & Play!
![Page 43: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/43.jpg)
43
Nimrod & Associated Family of Tools
P-sweep App. Composition: Nimrod/
EnfusionResource Management and Scheduling:
Nimrod-G BrokerDesign Optimisations:
Nimrod-OApp. Composition and Online Visualization:
Active SheetsGrid Simulation in Java:
GridSimDrug Design on Grid:
Virtual Lab
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
East
West
North
South
Remote Execution Server(on demand Nimrod Agent)
File Transfer Server
Upcoming?: HEPGrid (+U. Melbourne), GAVE(+Rutherford Appleton Lab)Grid (Un)Aware Virtual Engineering
![Page 44: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/44.jpg)
44
A resource broker for managing, steering, and executing task farming (parametric sweep/SPMD model) applications on Grid based on deadline and computational economy.
Based on users’ QoS requirements, our Broker dynamically leases services at runtime depending on their quality, cost, and availability.
Key Features A single window to manage & control experiment Persistent and Programmable Task Farming Engine Resource Discovery Resource Trading Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & results Steering & data management Accounting
Nimrod/G : A Grid Resource Broker
![Page 45: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/45.jpg)
45
A Glance at Nimrod-G Broker
Grid Middleware
Nimrod/G Client Nimrod/G ClientNimrod/G Client
Grid Information Server(s)
Schedule Advisor
Trading Manager
Nimrod/G Engine
GridStore
Grid Explorer
GE GISTM TS
RM & TS
Grid Dispatcher
RM: Local Resource Manager, TS: Trade Server
Globus, Legion, Condor, etc.
G
G
CL
Globus enabled node.Legion enabled node.
GL
Condor enabled node.
RM & TSRM & TS
C LSee HPCAsia 2000 paper!
![Page 46: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/46.jpg)
46
Globus Legion
Fabric
Nimrod Broker
Nimrod ClientsP-Tools (GUI/Scripting)(parameter_modeling)
Legacy Applications
P2P GTS
Farming Engine
Dispatcher & Actuators
Schedule Advisor
Trading Manager
Grid Explorer
Customised Apps(Active Sheet)
Monitoring and Steering Portals
Algorithm1
AlgorithmN
Middleware
. . .
Computers Storage Networks InstrumentsLocal Schedulers
G-Bank. . .
Agents
Resources
Programmable Entities Management
Jobs Tasks
. . .
AgentScheduler JobServer
PC/WS/Clusters Radio TelescopeCondor/LL/Mosix/ . . .Database
Meta-Scheduler
Nimrod/G Grid Broker Architecture
Globus-A
Channels
Legion-A P2P-A. . .
Database(Postgres)
XML
Condor GMD
XML?
IP hourglass ?
![Page 47: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/47.jpg)
47
A Nimrod/G Monitor
A Nimrod/G Monitor
CostCostDeadlineDeadline
Legion hosts
Globus Hosts
Bezek is in both Globus and Legion Domains
Arlington
Alexandria
Richmond
HamptonNorfolk
Virginia BeachChesapeakePortsmouth
Newport News
Roanoke
Ap p om a toxRive r
Ja m esRive r
Shena nd oa hRive r
Ra p p a ha nnoc kRive r
Potom a cRive r
VIRGINIA77
81
64
64
66
85
![Page 48: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/48.jpg)
48
User Requirements: Deadline/Budget User Requirements: Deadline/Budget
![Page 49: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/49.jpg)
49
Active Sheet: Spreadsheet Processing on Grid
NimrodNimrodProxyProxy
Nimrod/GNimrod/G
See HPC 2001 paper!
![Page 50: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/50.jpg)
50
![Page 51: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/51.jpg)
51
Nimrod/G Interactions
Grid Infoservers
Resource Discovery
QueuingSystem
Processserver
Resource allocation
(local)
Userprocess
File accessI/Oserver
Gatekeeper node
NimrodAgent
Computational node
Dispatcher
Root node
Scheduler
FarmingEngine
Grid Trade Server
“Do this in 30min. for $10?”
![Page 52: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/52.jpg)
52
Discover Discover ResourcesResources
Distribute JobsDistribute Jobs
Establish Establish RatesRates
Meet requirements ? Remaining Meet requirements ? Remaining Jobs, Deadline, & Budget ?Jobs, Deadline, & Budget ?
Evaluate & Evaluate & RescheduleReschedule
Discover Discover More More
ResourcesResources
Adaptive SchedulingAlgorithms
Execution Time (not beyond deadline)
Execution Cost (not beyond budget)
Time Minimisation Minimise Limited by budgetCost Minimisation Limited by deadline MinimiseNone Minimisation Limited by deadline Limited by budget
Adaptive Scheduling Algorithms
Compose & Compose & ScheduleSchedule
See HPDC AMS 2001 paper!
![Page 53: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/53.jpg)
53
Cost Model
Without cost ANY shared system becomes un-managable
Charge users more for remote facilities than their own
Choose cheaper resources before more expensive ones
Cost units (G$) may be Dollars Shares in global facility Stored in bank
![Page 54: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/54.jpg)
54
Cost Matrix @ Grid site X
Non-uniform costing Encourages use of
local resources first Real accounting
system can control machine usage
11 33
22 11User 5User 5
Mach
ine 1
Mach
ine 1
User 1User 1
Mach
ine 5
Mach
ine 5
Resource Cost = Function (cpu, memory, disk, network, software, QoS, current demand, etc.)
Simple: price based on peaktime, offpeak, discount when less demand, ..
![Page 55: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/55.jpg)
55
Deadline and Budget-based Cost Minimization Scheduling
1. Sort resources by increasing cost.2. For each resource in order, assign as
many jobs as possible to the resource, without exceeding the deadline.
3. Repeat all steps until all jobs are processed.
![Page 56: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/56.jpg)
56
M - Resources, N - Jobs, D - deadline Note: Cost of any Ri is less than any of Ri+1 …. Or Rm
RL: Resource List need to be maintained in increasing order of cost Ct - Time when accessed (Time now) Ti - Job runtime (average) on Resource i (Ri) [updated periodically]
Ti is acts as a load profiling parameter. Ai - number of jobs assigned to Ri , where:
Ai = Min (No.Unassigned Jobs, No. Jobs Ri can complete by remaining deadline) No.UnAssignedJobsi = Diff( N, (A1+…+Ai-1)) JobsRi consume = RemainingTime (D- Ct) DIV Ti
ALG: Invoke Job Assignment() periodically until all jobs done. Job Assignment()/Reassignment():
Establish ( RL, Ct , Ti , Ai ) dynamically – Resource Discovery. For all resources (I = 1 to M) { Assign Ai Jobs to Ri , if required}
Deadline-based Cost-minimization Scheduling
![Page 57: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/57.jpg)
57
Deadline and Budget Constraint (DBC) Time Minimization
Scheduling
1. For each resource, calculate the next completion time for an assigned job, taking into account previously assigned jobs.
2. Sort resources by next completion time.3. Assign one job to the first resource for
which the cost per job is less than the remaining budget per job.
4. Repeat all steps until all jobs are processed. (This is performed periodically or at each scheduling-event.)
![Page 58: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/58.jpg)
58
Deadline and Budget Constraint (DBC) Time+Cost Min. Scheduling
1. Split resources by whether cost per job is less than budget per job.
2. For the cheaper resources, assign jobs in inverse proportion to the job completion time (e.g. a resource with completion time = 5 gets twice as many jobs as a resource with completion time = 10).
3. For the dearer resources, repeat all steps (with a recalculated budget per job) until all jobs are assigned.
4. [Schedule/Reschedule] Repeat all steps until all jobs are processed.
![Page 59: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/59.jpg)
Evaluation of Scheduling Heuristics
A Hypothetical Application on
World Wide Grid
WW Grid
![Page 60: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/60.jpg)
60
Globus+LegionGRACE_TS
Australia
Monash Uni.:
Linux cluster
Solaris WS
Nimrod/G
Globus +GRACE_TS
Europe
ZIB/FUB: T3E/Mosix Cardiff: Sun E6500Paderborn: HPCLineLecce: Compaq SCCNR: ClusterCalabria: Cluster CERN: ClusterPozman: SGI/SP2
Globus +GRACE_TS
Asia/Japan
Tokyo I-Tech.:ETL, Tuskuba
Linux cluster
Globus/LegionGRACE_TS
North America
ANL: SGI/Sun/SP2USC-ISI: SGIUVa: Linux ClusterUD: Linux clusterUTK: Linux cluster
Internet
World Wide Grid (WWG)
Globus +GRACE_TS South America
Chile: Cluster
WW Grid
WW Grid
![Page 61: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/61.jpg)
61
Experiment-1 Setup
Workload: 165 jobs, each need 5 minute of cpu time
Deadline: 1 hrs. and budget: 800,000 units
Strategy: minimise cost and meet deadline
Execution Cost with cost optimisation AU Peaktime:471205 (G$) AU Offpeak time: 427155 (G$)
![Page 62: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/62.jpg)
62
Resources Selected & Price/CPU-sec.
Resource Type & Size
Owner and Location
Grid services
Peaktime Cost (G$)
Offpeak cost
Linux cluster (60 nodes)
Monash, Australia
Globus/Condor 20 5
IBM SP2 (80 nodes)
ANL, Chicago, US
Globus/LL 5 10
Sun (8 nodes) ANL, Chicago, US
Globus/Fork 5 10
SGI (96 nodes) ANL, Chicago, US
Globus/Condor-G
15 15
SGI (10 nodes) ISI, LA, US Globus/Fork 10 20
![Page 63: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/63.jpg)
63
Execution @ AU Peak Time
0
2
4
6
8
10
12
Time (minutes)
Jo
bs
Linux clus ter - Monash (20) Sun - ANL (5) SP2 - ANL (5) SGI - ANL (15) SGI - ISI (10)
![Page 64: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/64.jpg)
64
Execution @ AU Offpeak Time
0
2
4
6
8
10
12
Time (minutes)
Jo
bs
Linux clus ter - Monash (5) Sun - ANL (10) SP2 - ANL (10) SGI - ANL (15) SGI - ISI (20)
![Page 65: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/65.jpg)
65
AU peak: Resources/Cost in Use
0
50
100
150
200
250
300
350
400
450
500
Tim e (in m in.)
Co
st o
f R
eso
urc
es in
Use
0
5
10
15
20
25
30
35
40
Tim e (in m in.)
Res
ou
rces
(N
o. o
f C
PU
s) in
Use
After the calibration phase, note the difference in pattern of two graphs. This is when scheduler stopped using
expensive resources.
![Page 66: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/66.jpg)
66
AU offpeak: Resources/Cost in Use
0
50
100
150
200
250
300
350
Time (in min.)
Co
st o
f R
eso
urc
es i
n U
se
0
5
10
15
20
25
30
Time (in min.)
Res
ou
rces
(N
o.
of
CP
Us)
in
Use
![Page 67: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/67.jpg)
67
Experiment-2 Setup
Workload: 165 jobs, each need 5 minute of CPU time
Deadline: 2 hrs. and budget: 396000 units Strategy: minimise time / cost Execution Cost with cost optimisation
Optimise Cost: 115200 (G$) (finished in 2hrs.) Optimise Time: 237000 (G$) (finished in 1.25 hr.) In this experiment: Time-optimised scheduling run
costs double that of Cost-optimised. Users can now trade-off between Time Vs. Cost.
![Page 68: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/68.jpg)
68
Resources Selected & Price/CPU-sec.
Resource & Location
Grid services & Fabric
Cost/CPU sec.or unit
No. of Jobs Executed
Time_Opt Cost_Opt.
Linux Cluster-Monash, Melbourne, Australia
Globus, GTS, Condor
2 64 153
Linux-Prosecco-CNR, Pisa, Italy
Globus, GTS, Fork 3 7 1
Linux-Barbera-CNR, Pisa, Italy
Globus, GTS, Fork 4 6 1
Solaris/Ultas2
TITech, Tokyo, Japan
Globus, GTS, Fork 3 9 1
SGI-ISI, LA, US Globus, GTS, Fork 8 37 5
Sun-ANL, Chicago,US Globus, GTS, Fork 7 42 4Total Experiment Cost (G$) 237000 115200
Time to Complete Exp. (Min.) 70 119
![Page 69: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/69.jpg)
69
DBC Scheduling for Time Optimization
0
2
4
6
8
10
12
Time (in Minute)
No.
of
Tas
ks i
n E
xecu
tion
Condor-Monash Linux-Prosecco-CNR Linux-Barbera-CNR
Solaris /Ultas2-TITech SGI-ISI Sun-ANL
![Page 70: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/70.jpg)
70
DBC Scheduling for Cost Optimization
0
2
4
6
8
10
12
14
Time (in Minute)
No.
of
Tas
ks i
n E
xecu
tion
Condor-Monash Linux-Prosecco-CNR Linux-Barbera-CNR
Solaris /Ultas2-TITech SGI-ISI Sun-ANL
![Page 71: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/71.jpg)
Application Case Study
The Virtual Laboratory Project: "Molecular Modelling for Drug Design" on Peer-to-Peer Grid
![Page 72: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/72.jpg)
72
Virtual Drug Design: Data Intensive Computing on Grid
A Virtual Laboratory for “Molecular Modelling for Drug Design” on Peer-to-Peer Grid.
It provides tools for examining millions of chemical compounds (molecules) in the Protein Data Bank (PDB) to identify those having potential use in drug design.
In collaboration with: Kim Branson, Structural
Biology, Walter and Eliza Hall Institute (WEHI)
http://www.csse.monash.edu.au/~rajkumar/vlab
![Page 73: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/73.jpg)
73
Virtual Drug DesignA Virtual Lab for “Molecular Modeling for Drug Design” on P2P Grid
“Screen 2K molecules in 30min. for $10”
Grid Market Directory
ResourceBroker
Grid Info. Service
GTS
GTS
GTS
GTS
“Give me list PDBs sourcesOf type aldrich_300?”
“serv
ice co
st?”
(GTS - Grid Trade Server)
PDB2
“get mol.10 from pdb1 & screen it.”
Data Replica Catalogue
“service providers?”
GTS
PDB1
“mol.10 please?”
“mol.5 please?”
(RB maps suitable Grid nodes and Protein DataBank)
![Page 74: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/74.jpg)
74
DataGrid Brokering
Nimrod/GComputational
Grid Broker
Data Replica CataloguePDB Broker
Algorithm1
AlgorithmN
. . .
PDB Service
PDB2
“Screen mol.5 please?”
GSP1 GSP2 GSP4GSP3(Grid Service Provider)
GSPm
PDB Service
GSPn
1
“advise PDB source?
2“selection & advise: use GSP4!”
5Grid Info. Service
3
“Is GSP4 healthy?”
4
“mol.5 please?”6
“PDB replicas please?”
“Screen 2K molecules in 30min. for $10”
7
“process & send results”
![Page 75: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/75.jpg)
75
Software Tools
Molecular Modelling Tools (DOCK) Parameter Modelling Tools (Nimrod/enFusion) Grid Resource Broker (Nimrod-G) Data Grid Broker Protein Data Bank (PDB) Management and Intelligent Access
Tools PDB databse Lookup/Index Table Generation. PDB and associated index-table Replication. PDB Replica Catalogue (that helps in Resource Discovery). PDB Servers (that serve PDB clients requests). PDB Brokering (Replica Selection). PDB Clients for fetching Molecule Record (Data Movement).
Grid Middleware (Globus and GrACE) Grid Fabric Management (Fork/LSF/Condor/Codine/…)
![Page 76: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/76.jpg)
76
DOCK code*(Enhanced by WEHI, U of
Melbourne)
A program to evaluate the chemical and geometric complementarities between a small molecule and a macromolecular binding site.
It explores ways in which two molecules, such as a drug and an enzyme or protein receptor, might fit together.
Compounds which dock to each other well, like pieces of a three-dimensional jigsaw puzzle, have the potential to bind.
So, why is it important to able to identify small molecules which may bind to a target macromolecule?
A compound which binds to a biological macromolecule may inhibit its function, and thus act as a drug.
Thus disabling the ability of (HIV) virus attaching itself to molecule/protein!
With system specific code changed, we have been able to compile it for Sun-Solaris, PC Linux, SGI IRIX, Compaq Alpha/OSF1
* Original Code: University of California, San Francisco: http://www.cmpharm.ucsf.edu/kuntz/
![Page 77: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/77.jpg)
77
Dock input filescore_ligand yesminimize_ligand yesmultiple_ligands norandom_seed 7anchor_search notorsion_drive yesclash_overlap 0.5conformation_cutoff_factor 3torsion_minimize yesmatch_receptor_sites norandom_search yes . . . . . . . . . . . .maximum_cycles 1ligand_atom_file S_1.mol2receptor_site_file ece.sphscore_grid_prefix ecevdw_definition_file parameter/vdw.defnchemical_definition_file parameter/chem.defnchemical_score_file parameter/chem_score.tblflex_definition_file parameter/flex.defnflex_drive_file parameter/flex_drive.tblligand_contact_file dock_cnt.mol2ligand_chemical_file dock_chm.mol2ligand_energy_file dock_nrg.mol2
Molecule to Molecule to be screenedbe screened
![Page 78: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/78.jpg)
78
score_ligand $score_ligandminimize_ligand $minimize_ligandmultiple_ligands $multiple_ligandsrandom_seed $random_seedanchor_search $anchor_searchtorsion_drive $torsion_driveclash_overlap $clash_overlapconformation_cutoff_factor $conformation_cutoff_factortorsion_minimize $torsion_minimizematch_receptor_sites $match_receptor_sitesrandom_search $random_search . . . . . . . . . . . .maximum_cycles $maximum_cyclesligand_atom_file ${ligand_number}.mol2receptor_site_file $HOME/dock_inputs/${receptor_site_file}score_grid_prefix $HOME/dock_inputs/${score_grid_prefix}vdw_definition_file vdw.defnchemical_definition_file chem.defnchemical_score_file chem_score.tblflex_definition_file flex.defnflex_drive_file flex_drive.tblligand_contact_file dock_cnt.mol2ligand_chemical_file dock_chm.mol2ligand_energy_file dock_nrg.mol2
Parameterized Dock input file
Molecule to be Molecule to be screenedscreened
![Page 79: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/79.jpg)
79
parameter database_name label "database_name" text select oneof "aldrich" "maybridge" "maybridge_300" "asinex_egc" "asinex_epc" "asinex_pre" "available_chemicals_directory" "inter_bioscreen_s" "inter_bioscreen_n" "inter_bioscreen_n_300" "inter_bioscreen_n_500" "biomolecular_research_institute" "molecular_science" "molecular_diversity_preservation" "national_cancer_institute" "IGF_HITS" "aldrich_300" "molecular_science_500" "APP" "ECE" default "aldrich_300";
parameter score_ligand text default "yes";parameter minimize_ligand text default "yes";parameter multiple_ligands text default "no";parameter random_seed integer default 7;parameter anchor_search text default "no";parameter torsion_drive text default "yes";parameter clash_overlap float default 0.5;parameter conformation_cutoff_factor integer default 5;parameter torsion_minimize text default "yes";parameter match_receptor_sites text default "no";parameter random_search text default "yes"; . . . . . . . . . . . .parameter maximum_cycles integer default 1;parameter receptor_site_file text default "ece.sph";parameter score_grid_prefix text default "ece";parameter ligand_number integer range from 1 to 200 step 1;
Dock PlanFile (contd.)
Molecules to be Molecules to be screenedscreened
![Page 80: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/80.jpg)
80
task nodestart copy ./parameter/vdw.defn node:. copy ./parameter/chem.defn node:. copy ./parameter/chem_score.tbl node:. copy ./parameter/flex.defn node:. copy ./parameter/flex_drive.tbl node:. copy ./dock_inputs/get_molecule node:. copy ./dock_inputs/dock_base node:.endtasktask main node:substitute dock_base dock_run node:substitute get_molecule get_molecule_fetch node:execute sh ./get_molecule_fetch node:execute $HOME/bin/dock.$OS -i dock_run -o dock_out copy node:dock_out ./results/dock_out.$jobname copy node:dock_cnt.mol2 ./results/dock_cnt.mol2.$jobname copy node:dock_chm.mol2 ./results/dock_chm.mol2.$jobname copy node:dock_nrg.mol2 ./results/dock_nrg.mol2.$jobnameendtask
Dock PlanFile
![Page 81: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/81.jpg)
81
Nimrod/TurboLinux enFuzion GUI tools for Parameter Modeling
![Page 82: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/82.jpg)
82
Docking Experiment Preparation
Setup PDB DataGrid Index PDB databases Pre-stage (all) Protein Data Bank (PDB) on replica sites Start PDB Server
Create Docking GridScore (receptor surface details) for a given receptor on home node.
Pre-Staging Large Files required for Docking: Pre-stage Dock executables and PDB access client on Grid nodes, if
required (e.g., dock.Linux, dock.SunOS, dock.IRIX64, and dock.OSF1 on Linux, Sun, SGI, and Compaq machines respectively). Use globus-rcp.
Pre-stage/Cache all data files (~3-13MB each) representing receptor details on Grid nodes.
This can can be done demand by Nimrod/G for each job, but few input files are too large and they are required for all jobs). So, pre-staging/caching at http-cache or broker level is necessary to avoid the overhead of copying the same input files again and again!
![Page 83: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/83.jpg)
83
Protein Data Bank
Databases consist of small molecules from commercially available organic synthesis libraries, and natural product databases.
There is also the ability to screen virtual combinatorial databases, in their entirety.
This methodology allows only the required compounds to be subjected to physical screening and/or synthesis reducing both time and expense.
![Page 84: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/84.jpg)
84
Target Testcase
The target for the test case: electrocardiogram (ECE) endothelin converting enzyme. This is involved in “heart stroke” and other transient ischemia.
Is·che·mi·a : A decrease in the blood supply to a bodily organ, tissue, or part caused by constriction or obstruction of the blood vessels.
![Page 85: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/85.jpg)
85
Nimrod/G in Action:Screening on World-Wide
Grid
![Page 86: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/86.jpg)
86
Any Scientific Discovery ? Did your collaborator invent new drug for
xxxx?
Not Yet
Anyway, checkout the announcement of Nobel-
prize winners for next year
?
![Page 87: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/87.jpg)
Conclude with a comparison with the Electrical
Grid………..
Where we are ????
Courtesy: Domenico Laforenza
![Page 88: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/88.jpg)
Alessandro Volta in Paris in 1801 inside French National Institute shows the battery
while in the presence of Napoleon I
Fresco by N. Cianfanelli (1841) (Zoological Section "La Specula" of National History Museum of Florence
University)
![Page 89: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/89.jpg)
89
….and in the future, I imagine a worldwidePower (Electrical) Grid …...
What ?!?!This is a mad man…
Oh, monDieu !
![Page 90: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/90.jpg)
90
2001 - 1801 = 200 Years
![Page 91: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/91.jpg)
91
” I think there is a world market for about five computers.”Thomas J. Watson Sr., IBM Founder, 1943
Can we Predict its Future ?
![Page 92: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/92.jpg)
92
Summary and Conclusions P2P and Grid Computing is emerging as a next generation
computing platform for solving large scale problems through sharing of geographically distributed resources.
Resource management is a complex undertaking as systems need to be adaptive, scalable, competitive,…, and driven by QoS.
We proposed a framework based on “computational economies” and discussed several economic models for resource allocation and for regulating supply-and-demand for resources.
Scheduling experiments on World Wide Grid demonstrate our Nimrod-G broker ability to dynamically lease or rent services at runtime based on their quality, cost, and availability depending on consumers QoS requirements.
Economics paradigm for QoS driven resource management is essential to push P2P/Grids into mainstream computing!
![Page 93: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/93.jpg)
93
Download Software & Information
Nimrod & Parameteric Computing: http://www.csse.monash.edu.au/~davida/nimrod/
Economy Grid & Nimrod/G: http://www.buyya.com/ecogrid/
Virtual Laboratory/Virtual Drug Design: http://www.buyya.com/vlab/
Grid Simulation (GridSim) Toolkit (Java based): http://www.buyya.com/gridsim/
World Wide Grid (WWG) testbed: http://www.buyya.com/ecogrid/wwg/ Looking for new volunteers to grow
Please contact me to barter your & our machines!
Want to build on our work/collaborate: Talk to me now or email: [email protected]
![Page 94: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/94.jpg)
94
Thank You… Any ??
Thank You… Any ??
![Page 95: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/95.jpg)
95
Further Information
Books: High Performance Cluster Computing, V1,
V2, R.Buyya (Ed), Prentice Hall, 1999. The GRID, I. Foster and C. Kesselman (Eds),
Morgan-Kaufmann, 1999. IEEE Task Force on Cluster Computing
http://www.ieeetfcc.org Global Grid Forum
www.gridforum.org
IEEE/ACM CCGrid’xy: www.ccgrid.org CCGrid 2002, Berlin: ccgrid2002.zib.de
Grid workshop - www.gridcomputing.org
![Page 96: Economics Paradigm for Resource Management and Scheduling for Service Oriented P2P/Grid Computing Rajkumar Buyya Melbourne, Australia](https://reader033.vdocument.in/reader033/viewer/2022061306/55146472550346494e8b5ab7/html5/thumbnails/96.jpg)
96
Further Information
Cluster Computing Info Centre: http://www.buyya.com/cluster/
Grid Computing Info Centre: http://www.gridcomputing.com
IEEE DS Online - Grid Computing area:
http://computer.org/dsonline/gc
Compute Power Market Project http://www.ComputePower.com