grid tools
DESCRIPTION
Grid Tools. Jai Dayal & Casey Rathbone. Outline. Web Services Unicore BOINC Globus Condor. Web Services. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/1.jpg)
Grid Tools
Jai Dayal & Casey Rathbone
![Page 2: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/2.jpg)
Outline Web Services Unicore BOINC Globus Condor
![Page 3: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/3.jpg)
Web Services“A Web service is an abstract notion that must be implemented by a concrete agent. … The agent is the concrete piece of software or hardware that sends and receives messages, while the service is the resource characterized by the abstract set of functionality that is provided.”
![Page 4: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/4.jpg)
Web Services
![Page 5: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/5.jpg)
Web Services
Remote Procedure Calls (RPC) - distributed function call interface
Service Oriented Architecture (SOA) - utilizes message passing to communicate to services also known as "message-oriented" services
Representational State Transfer (REST) - emulates HTTP (or similar protocols) to communicate stateful resources
![Page 6: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/6.jpg)
Unicore - Overview Single point of entry – security
Uses SSL
Uses web services SOAP Messages WSDL
Reputation for simple deployment
![Page 7: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/7.jpg)
Unicore - Architecture
Most basic resourceA cluster, a PC, web service, etc
Vsite
![Page 8: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/8.jpg)
Unicore - Architecture The gateway is the point of entry to the
Unicore grid
Vsites
Gateway
![Page 9: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/9.jpg)
Unicore - Architecture Gateway + Vsites = Usite
Vsites
Gateway
Usite
![Page 10: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/10.jpg)
Unicore - ArchitectureUNICORE - Architecture
Vsites
Gateway
Usite
Client
![Page 11: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/11.jpg)
Unicore - Architecture
Vsites
Gateway
Usite
Client
Client communicates using SOAP SOAP/HTTP binding
SOAP Message
SOAP contains: Address of
gateway Port number Vsite name task descriptions
![Page 12: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/12.jpg)
Unicore - Security• Gateway handles authentication
– SSL used to authenticate the client– SSL also used by client to authenticate
gateway– Both present digital certificates
• Vsite handles authorization– Gateway passes along certificate in SOAP header– If client has privileges, Vsite authorizes
•This model allows end to end security
• Removes some strain from gateway
• Trust between gateway and Vsite
![Page 13: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/13.jpg)
BOINC - Overview
Grid computing vs. public resource computing
Grid Public Resource Computing
Owned by institutions/orgs Individuals own pc's
Managed/used by pro's Used by commoners
High level of trust Low (no?) level of trust
![Page 14: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/14.jpg)
BOINC - Overview
Grid computing vs. public resource computing
Grid Public Resource Computing
Owned by institutions/orgs Individuals own pc's
Managed/used by pro's Used by commoners
High level of trust Low (no?) level of trust
BOINC GoalsCheap and easyProject varietyPublic participation → reward points
![Page 15: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/15.jpg)
BOINC - Overview Grid computing vs. public resource computing
Grid Public Resource Computing
Owned by institutions/orgs Individuals own pc's
Managed/used by pro's Used by commoners
High level of trust Low (no?) level of trust BOINC Goals
Cheap and easyProject varietyPublic participation → reward points
BOINC projectsSETI at homeFolding at homePredictor at home
![Page 16: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/16.jpg)
BOINC - ArchitectureBOINC
![Page 17: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/17.jpg)
BOINC - ArchitectureBOINCProjects
![Page 18: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/18.jpg)
BOINC - ArchitectureBOINCProjects
Servers
Scheduling Data
![Page 19: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/19.jpg)
BOINC - ArchitectureBOINCProjects
Servers
Scheduling Data
Tools
![Page 20: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/20.jpg)
BOINC - ArchitectureBOINCProjects
Servers
Scheduling Data
Tools
Client
![Page 21: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/21.jpg)
BOINC - Workflow Client contacts Scheduling Server
Server returns XML Document
Workunit: inputs for a computation/jobApplicationsInput filesEnvironment variables
Client runs job and places results in output files
Result:Reference to workunitReference to output files
![Page 22: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/22.jpg)
BOINC – Fault tolerance
TransitionerIdentifies bad results
ValidatorDetermines canonical result
Public Resource Computing is volatileUsers leave before job completingOver-clocking can lead to bad resultsMalicious users
AssimilatorHandles canonical result
These daemons can run on different hosts
Homogenous RedundancyComputer architecture specific jobs
![Page 23: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/23.jpg)
BOINC – Client settings Client implements “local scheduling policy”
Utilizes maximum resourcesMeet job deadlinesRespect clients resource allocation wishesOffers some variety to the client
Anonymous platformUsed for open source projectsClients can modify apps to fit their resourcesCommunicate with scheduling server using XML config file.
![Page 24: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/24.jpg)
Globus Toolkit 4
• GT4 is a open-source toolkit developed to build grids • It provides full capabilities for sharing computing power
and databases • Usage of Globus is extensive throughout the scientific
community within NSF, DOE, DARPA, IBM, NASA, and Microsoft projects
• The Large Hadron Collider project utilizes a Globus based grid for the European Data Grid.
![Page 25: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/25.jpg)
Globus Toolkit 4
Globus Toolkit is a suite of Web-Services and Unix that controls provides an interface to submit, monitor, and cancel jobs on the Grid resources.
Globus Toolkit is an interface that allows for jobs to be submitted and executed at the appropriate resources within the grid
![Page 26: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/26.jpg)
Globus Toolkit 4
![Page 27: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/27.jpg)
Globus Toolkit 4 Security• GT4 uses (Grid Security Infrastructure) GSI with an public
key cryptography • GSI Functional Layers - Authorization, Authentications,
Delegation, Message Protection, and Message Format
![Page 28: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/28.jpg)
Globus Toolkit 4 Data Management
• Data Management tools within the toolkit fall into either of two categories data replication and data movement
• Data Replication consists of Replica Location Service (RLS) • Data Movement consists of GridFTP and Reliable File
Transfer (RFT)
![Page 29: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/29.jpg)
GT4: Data Replication
Replica Location Service provides the capability to track and maintain multiple locations of data across the grid. It is a distributed registry system that allows users and applications to register the locations of data.
![Page 30: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/30.jpg)
GT4: Data Movement
• GridFTP - GT4 provides a server implementation, scriptable command-line interface and several GUI client interfaces
• Reliable File Transfer (RFT) - A Web Services Resource
Framework that schedules file transfers based on a set of criteria of when specific resources and bandwidth is available
![Page 31: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/31.jpg)
Globus Toolkit 4 Execution Management
• Community Scheduler Framework - Grid level meta-scheduling framework
• Grid Teleoperation Control Protocol - A service interface that has been used to
• Dynamic Accounts - This service provides functionality to create groups and accounts for specific jobs that need to be submitted
• Grid Resource Allocation & Management - Used to locate, submit, monitor and cancel jobs on Grid computing resources. It is the fabric that ties all the schedulers and job queues together to utilize the resources.
![Page 32: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/32.jpg)
GT4: Community Scheduler Framework
Open-Source implementation of OSGA-based meta scheduler
Provides an interface to existing grid platforms making integration more efficient
![Page 33: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/33.jpg)
GT4: Community Scheduler Framework
The diagram below shows the basic layout of the scheduler framework
http://www.globus.org/grid_software/computation/csf.php
![Page 34: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/34.jpg)
GT4: Grid Teleoperation Control Protocol
The GTCP is used to control heterogenous physical and computational simulations.It exposes the WSRF-compliant service interface used to control remote instruments and simulations and a interface to the backend GTCP to allow for additional control abstraction.
![Page 35: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/35.jpg)
Globus Toolkit 4 Information Services
Monitoring and Discovery Service is a suite of web services used to discover resources and services on a grid
This service is dedicated to handling the complex task of organizing resources into Virtual Organizations (VO)
http://www.globus.org/toolkit/docs/4.0/info/key-index.html
![Page 36: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/36.jpg)
Globus Toolkit 4 Common Runtime
GT4 contains a multitude of implementations of libraries allowing the suite to be platform independent.
• C Common Libraries• C WebService Libraries• Java WebService Libraries• Python WebService Libraries• XIO
![Page 37: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/37.jpg)
Condor - Overview• University of Wisconsin – Madison• Open source• Linux, Solaris, Mac OS, Windows• No program modification
• Jobs tend to be architecture specific– Designed on x86 → job run on x86– Java applications possible exception
• Overall concept is simple– Install it on each desired machine– Designate one as central manager
• Practice is much harder– Installation is tricky– Lots of commands to learn
![Page 38: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/38.jpg)
Condor• Central Manager
– Detects idle machines– Matches job with available resources– Considers each processor as separate resource
• Jobs– Submitted via “submit file”– Universe, executable name, inputs, outputs, error, log file
• Sequential Jobs– Standard universe
• Used for long jobs (weeks, months)• Allows partial execution, pausing• Can relocate job to another machine
– Vanilla universe• For short term jobs
– Local universe• Submitter executes job
• Parallel Jobs– Supports MPI and PVM
![Page 39: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/39.jpg)
Condor - variety• Flocking
– Connects multiple pools– If current pool has no resources, job “flocks”– Set “flock_to” variable in config file
• Condor-G– Distributed flocking– Utilizes Globus– Universe = Globus– Also specify “globusscheduler”
• Points to the globus job manager
![Page 40: Grid Tools](https://reader035.vdocument.in/reader035/viewer/2022081603/568146fd550346895db43807/html5/thumbnails/40.jpg)
References[1] D. P. Anderson, BOINC: A System for Public-Resource Computing and Storage. In Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing. November, 2004.
[2] S. Bagchi, E. Hung, A. Iyengar, N. Vogl, and N. Wadia, Capacity Planning Tools for Web and Grid Environments. In Proceedings of the 1st international conference on Performance evaluation methodolgies and tools. October, 2006.
[3] F. Hernández, P. Bangalore, K. Reilly, End-User Tools for Grid Computing. In Proceedings of the first workshop on End-user software engineering. July, 2005.
[4]I. Habib, Getting Started with Condor. Linux Journal, Volume 2006, Issue 149, September 2006.
[5]Menday, R., "The Web Services Architecture and the UNICORE Gateway," Telecommunications, 2006. AICT-ICIW '06. International Conference on Internet and Web Applications and Services/Advanced International Conference on , vol., no., pp. 134-134, 19-25 Feb. 2006
[6]Hellinckx, P.; Stuer, G.; Hendrickx, W.; Arickx, F.; Broeckhove, J., "Grid-user driven grid research, the CoBRA grid," Cluster Computing and the Grid Workshops, 2006. Sixth IEEE International Symposium on , vol.2, no., pp. 8 pp.-, 16-19 May 2006