![Page 1: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/1.jpg)
Rochester Institute of Technology
Cyberaide Shell: Interactive Task Management for Grids and
Cyberinfrastructure
Gregor von Laszewski, Andrew J. Younge, Xi He, and Fugang Wang
Service Oriented Cyberinfrastructure Laboratory
Rochester Institute of Technology
Bldg 74, Lomb Memorial Drive
Rochester, NY 14523-5608
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 1
![Page 2: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/2.jpg)
Agenda
• Introduction
• Related Research
• Problem
• Proposed Solution
• Design
• Implementation
• Use Case
• Conclusion
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 2
![Page 3: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/3.jpg)
Introduction
What is Grid Computing?
“Grid computing is a complex and diverse field where different technologies are constructed and combined to enable the use of distributed resources under administratively separate domains.”
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 3
![Page 4: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/4.jpg)
Grid Hierarchy
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 4
![Page 5: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/5.jpg)
Grid Fabric (Clusters)
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 5
All clusters are batch queuing systems
![Page 6: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/6.jpg)
Middleware• Middleware combines multiple clusters together.
– Administratively separate systems– Large dedicated clusters
• Globus Toolkit– Most popular of all Grid middleware technologies
• Developed at Argonne National Laboratory• GT4 uses advanced WSRF Web Services
• G-Lite– Developed by the EGEE project– Used in the Large Hadron Collider experiments
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 6
![Page 7: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/7.jpg)
The TeraGrid NSF-funded national-scale Grid Infrastructure
11 Locations – LONI, NCAR, NCSA, NICS, ORNL, PSC, IU, PU, SDSC, TACC, UC/ANL
1.1Petaflops, 161 thousand CPUs, 60 Petabytes disk space Dedicated 10G fiber lines to each location Specialized visualization servers
Uses Globus Toolkit 4’s basic WS services and security protocols
Grid Infrastructure Group (GIG) at U. Chicago Commity for Teragrid planning, management, and coordination
Science Gateways Independent services for specialized groups and organizations “TeraGrid Inside” capabilities Web Portals, desktop apps, coordinated access points Not Virtual Organizations (VOs)
![Page 8: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/8.jpg)
TeraGrid View
SDSC
TACC
UC/ANL
NCSA
ORNL
PU
IU
PSC
NCAR
CaltechUNC/RENCI
UW
Resource Provider (RP)
Software Integration Partner
Grid Infrastructure Group (UChicago)
LONI
NICS
![Page 9: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/9.jpg)
TeraGrid User Portal
![Page 10: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/10.jpg)
Introduction
What is a System Shell?
“A system shell is a piece of software that provides a text based interface which abstracts complex services for users.”
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 10
![Page 11: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/11.jpg)
Shell environments
• Shells are used to abstract complex services
• Typically used in Operating Systems• Hide the complexity of the underlying kernel and
support services• Provide a unified user environment
• Command-Line Interface– To many power users, this is preferred– More function and control available
• Shells can be scripted to automate common high-level tasks.
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 11
![Page 12: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/12.jpg)
Shells used
• System shells are used as part of an OS to provide a CLI for users
• Distributed Shells execute tasks similar to system shells, however in Parallel
• Language Shells interprets users’s code and commands during runtime
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 12
![Page 13: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/13.jpg)
Problem
• A steep learning curve and high entry barrier limit the use of Grid computing within the scientific community.– Parallel programming is difficult – Using middleware technologies can be
cumbersome and inefficient– Users need to be advanced programmers just to
use the technology
• How do we simplify the Cyberinfrastructure environment for scientists?
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 13
![Page 14: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/14.jpg)
Requirements
• Simple, intuitive, and user friendly
• Unifying framework
• Extensible
• Interactive
• Fault tolerant
• Event management
• Support batch processing
• Integrates new Cyberinfrastructure
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 14
![Page 15: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/15.jpg)
What if we combined Grids with Shells?
Performance + Usability
(have your cake and eat it too)
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 15
![Page 16: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/16.jpg)
Cyberaide Shell
• Simple to use system shell which provides access to the powerful Grids and advanced Cyberinfrastructure that is available today
• However, “This isn’t your father’s UNIX shell…”
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 16
![Page 17: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/17.jpg)
High Level Design
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 17
![Page 18: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/18.jpg)
Architectural Design
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 18
![Page 19: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/19.jpg)
Implementation
– Set– Submit– Person– Man
– List– Experiment– Resources– Tasks
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 19
• Prototype implementation
• Demonstrates our design’s feasibility
• Uses Globus GT2 and GT4 technologies for backend services
• Supports basic commands
![Page 20: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/20.jpg)
Command List
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 20
![Page 21: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/21.jpg)
Nested Shells
• Cyberaide Shell supports the idea of Nested Shells– “shell-within-a-shell”– Each command has its own interpreter which
interrogates options dynamically– More efficient scripts
• Two Ways:
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 21
![Page 22: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/22.jpg)
Literate Semantic Objects (LSOs)
• New data management concept• LSOs store attribute-value pairs – simple!• Mutable during runtime• Object inheritance
– All objects inherent form this object
• Can be recorded in many different formats– XML, JSON, CSV– Converted to SQL
• Provide semantic information to the shell during runtime
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 22
![Page 23: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/23.jpg)
Extending Cyberaide Shell
• Creating a Service Oriented Architecture
• Expose shell a Web Service– Web Services Interoperability standar– WS-Security for endpoint to endpoint security with
X.509 certificates– Uses Apache CXF and Jetty HTTP server to host
Web service
• Various Client API’s possible
• Using Web service, its possible to build a Web Portal on top of the shell!
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 23
![Page 24: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/24.jpg)
Cyberaide Portal
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 24
![Page 25: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/25.jpg)
Use Case
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 25
![Page 26: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/26.jpg)
Future Work
• Implement other Cyberinfrastructure services– EGEE project and the Open Science Grid– Amazon Elastic Compute Cloud (EC2)
• Improve scheduling and workflow systems
• Improving integration with the Portal
• Develop language specific APIs
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 26
![Page 27: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/27.jpg)
Conclusion
• Cyberaide Shell helps scientists overcome the challenges faced when using advanced Cyberinfrastructure
• Combines the power of Grids with the usability of shells
• Enhances previous work in system shells with the use of semantic commands and nested commands
• Provides the shell as a Web service to allow for distributed usage across multiple platforms
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 27
![Page 28: Cyberaide Shell: Interactive Task Management for Grids and Cyberinfrastructure](https://reader030.vdocument.in/reader030/viewer/2022032605/56812b62550346895d8f8481/html5/thumbnails/28.jpg)
04/19/23 Service Oriented Cyberinfrastructure Lab, http://grid.rit.edu 28
Questions?