![Page 1: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/1.jpg)
1CEOS WGISS Meeting, May 8-12, 2006, Budapest
MTA SZTAKIComputer and Automation Research Institute
Hungarian Academy of Sciences
www.lpds.sztaki.hu/[email protected]
Grid Application SupportGrid Application Support by the P-GRADE Portal by the P-GRADE Portal
Gergely SiposGergely Sipossipossipos@@sztaki.husztaki.hu
![Page 2: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/2.jpg)
2CEOS WGISS Meeting, May 8-12, 2006, Budapest
OutlineOutline
• Motivation of our work
• P-GRADE Portal in a nutshell
• Grid application development
• Grid application execution and monitoring
• Support for collaborative work
• Conclusions
![Page 3: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/3.jpg)
3CEOS WGISS Meeting, May 8-12, 2006, Budapest
Layered view of Grid systemsLayered view of Grid systems
Basic Grid services:security, job submission, info, …
Higher-level grid services: brokering, data replica, …
Application toolkits, portals
Applications
Command line grid tools
Graphical grid tools:P-GRADE Portal
Resources to be shared
Grid middleware:e.g. Globus, EGEE, …
E-Scientists
![Page 4: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/4.jpg)
4CEOS WGISS Meeting, May 8-12, 2006, Budapest
Motivation to theMotivation to theP-GRADE PortalP-GRADE Portal
• Fast evolution of Grid middleware technologies and tools:– GT2, OGSA, GT3 (OGSI), GT4 (WSRF), LCG-2, gLite, …
• Many production Grids are now freely available for e-Scientists– EGEE (LCG-2 gLite), UK NGS (GT2),
US Open Science Grid (GT2 GT4), NorduGrid (ARC), …• The same set of services are available everywhere, but
implemented in different ways– Computation services, data services, security services, (brokers)
Let’s provide a technology-neutral, intuitive graphical interface for the most common Grid middleware services!
![Page 5: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/5.jpg)
5CEOS WGISS Meeting, May 8-12, 2006, Budapest
P-GRADE Portal in a nutshellP-GRADE Portal in a nutshell• General purpose, workflow-oriented computational Grid
portal. Supports the development and execution of workflow-based Grid applications.
• Based on standard portlet framework (Gridsphere)– Easy to expand with new portlets (e.g. application-specific portlets)– Easy to tailor to community needs
• Grid services supported by the portal:
Service EGEE grids Globus grids
Job execution Computing Element GRAM
File storage Storage Element GridFTP server
Certificate management MyProxy
Information system BDII MDS-2
Brokering Workload Management System ---
Job monitoring Mercury
Workflow & job visualization PROVE
The P-GRADE Portal hides middleware technologies and solves Grid interoperability problem at the workflow level
![Page 6: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/6.jpg)
6CEOS WGISS Meeting, May 8-12, 2006, Budapest
What is a P-GRADE Portal workflow?What is a P-GRADE Portal workflow?
• a directed acyclic graph where– Nodes represent jobs (batch
programs to be executed on a computing resource)
– Ports represent input/output files the jobs expect/produce
– Arcs represent file transfer operations
• semantics of the workflow:– A job can be executed if all
of its input files are available
![Page 7: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/7.jpg)
7CEOS WGISS Meeting, May 8-12, 2006, Budapest
Two levels of parallelismTwo levels of parallelism
• The workflow concept of the P-GRADE Portal enables the efficient parallelization of complex problems
• Semantics of the workflow enables two levels of parallelism:
Each job can be a parallel program
– Parallel execution inside a workflow node– Parallel execution among workflow nodes
Multiple jobs run parallel
![Page 8: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/8.jpg)
8CEOS WGISS Meeting, May 8-12, 2006, Budapest
25 x
10 x25 x 5 x
Forecasting dangerous weather situations (storms, fog, etc.), crucial task in the protection of life and property
Processed information:surface level measurements, high-altitude measurements, radar, satellite, lightning, results of previous computed models
Requirements:•Execution time < 10 min•High resolution (1km)
Ultra-short range weather forecast Ultra-short range weather forecast (Hungarian Meteorology Service)(Hungarian Meteorology Service)
![Page 9: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/9.jpg)
9CEOS WGISS Meeting, May 8-12, 2006, Budapest
Multi-Grid portal: Multi-Grid portal: Demonstration @ EGEE User ForumDemonstration @ EGEE User Forum
Job Repository Job Repository (GEMLCA)(GEMLCA)
ManchesterPortal Client in Geneva
Leeds
P-GRADE Portalserver
Portal Server in London
Executable
NGS GT2 (UK)
WestFocus GT4 Grid (UK)
UoW
Brunel
Poznan
Budapest
EGEE (CE)
Executable
Executable
ServiceInvocation
Executable
![Page 10: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/10.jpg)
10CEOS WGISS Meeting, May 8-12, 2006, Budapest
The typical user scenarioThe typical user scenarioPart 1 - development phasePart 1 - development phase
Certificate servers
Portalserver
Gridservices
START EDITOR
OPEN & EDIT or DEVELOP WORKFLOW
SAVE WORKFLOW
![Page 11: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/11.jpg)
11CEOS WGISS Meeting, May 8-12, 2006, Budapest
Certificate servers
Portalserver
Gridservices
TRANSFER FILES, SUBMIT JOBS
DOWNLOAD RESULTS
DOWNLOAD RESULTS
The typical user scenarioThe typical user scenarioPart 2 - execution phasePart 2 - execution phase
VISUALIZE JOBS and
WORKFLOW PROGRESS
MONITOR JOBS
DOWNLOAD PROXY CERTIFICATES
SUBMIT WORKFLOW
![Page 12: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/12.jpg)
12CEOS WGISS Meeting, May 8-12, 2006, Budapest
Opening the workflow editorOpening the workflow editor
The editor is a Java Webstart applicationdownload and installation is only a click!
![Page 13: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/13.jpg)
13CEOS WGISS Meeting, May 8-12, 2006, Budapest
Workflow Workflow EditorEditor
Define DAG of batch jobs:1. Drag & drop components:
jobs and ports
2. Connect ports by channels (no cycles, no loops)
3. Define job and port properties
![Page 14: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/14.jpg)
14CEOS WGISS Meeting, May 8-12, 2006, Budapest
Job property windowJob property window
Properties of a job:• Binary executable• (Type of executable)• (Number of processors)• Command line
parameters• The resource to be used
for the execution:•Grid•Broker / resource
![Page 15: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/15.jpg)
15CEOS WGISS Meeting, May 8-12, 2006, Budapest
Support for manual resource selection:Support for manual resource selection:information system browserinformation system browser
The information system portlet queries EGEE
and Globus information systems
![Page 16: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/16.jpg)
16CEOS WGISS Meeting, May 8-12, 2006, Budapest
Support for broker-based Support for broker-based resource selectionresource selection
1. Select a broker Grid/VO for the job
2. (Specify extra ranks & requirements for the job in Job Description Language)
3. The broker will find the best resource for your job!
![Page 17: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/17.jpg)
17CEOS WGISS Meeting, May 8-12, 2006, Budapest
Defining input/output dataDefining input/output data
File type: input: required by the job output: produced by the jobFile location: local: my desktop remote: grid storage siteFile name: unique name of the fileFile storage type: Permanent: final result of the WF Volatile: only used for inter-job data transfer
![Page 18: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/18.jpg)
18CEOS WGISS Meeting, May 8-12, 2006, Budapest
The typical user scenarioThe typical user scenarioExecution phase – step 1:Execution phase – step 1:
Certificate servers
Portalserver
Gridservices
DOWNLOAD PROXY CERTIFICATES
![Page 19: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/19.jpg)
19CEOS WGISS Meeting, May 8-12, 2006, Budapest
Prerequisite of workflow execution: Prerequisite of workflow execution: Download user certificatesDownload user certificates
1. Certificate server details:• Hostname• Port number• User name• Password
2. Certificate parameters:• Lifetime
![Page 20: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/20.jpg)
20CEOS WGISS Meeting, May 8-12, 2006, Budapest
Result of certificate downloadResult of certificate download
Multiple certificates for multiple Grids/VOs can be available on the portal server at the same time!
SEE-GRID resourcesHUNGRID resources
![Page 21: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/21.jpg)
21CEOS WGISS Meeting, May 8-12, 2006, BudapestWhite/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
![Page 22: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/22.jpg)
22CEOS WGISS Meeting, May 8-12, 2006, BudapestWhite/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
![Page 23: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/23.jpg)
23CEOS WGISS Meeting, May 8-12, 2006, BudapestWhite/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
![Page 24: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/24.jpg)
24CEOS WGISS Meeting, May 8-12, 2006, BudapestWhite/Red/Green color means the job is initial/running/finished state
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
![Page 25: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/25.jpg)
25CEOS WGISS Meeting, May 8-12, 2006, Budapest
Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)
White/Red/Green color means the job is initialised/running/finished
![Page 26: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/26.jpg)
26CEOS WGISS Meeting, May 8-12, 2006, Budapest
On-Line application monitoringOn-Line application monitoring
- The portal monitors and visualizes parallel jobs
- The portal monitors and visualizes workflows:
-Input data transfer
-Job execution
-Output data transfer
![Page 27: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/27.jpg)
27CEOS WGISS Meeting, May 8-12, 2006, Budapest
Downloading the results…Downloading the results…
- Download small files in a single ZIP file
- Keep large files in the Grid for future analysis!
![Page 28: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/28.jpg)
28CEOS WGISS Meeting, May 8-12, 2006, Budapest
Collaborative work Collaborative work by the P-GRADE Portalby the P-GRADE Portal
• Export and import workflows and execution results from/into the portal
• Share legacy applications as services with other portal users• Share successfully executed jobs as services with other
portal users• Integrate other users’ legacy applications and jobs into your
workflow
![Page 29: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/29.jpg)
29CEOS WGISS Meeting, May 8-12, 2006, Budapest
• P-GRADE Portal service is available for– SEE-GRID infrastructure– HUNGRID VO of EGEE– Central European VO of EGEE– GILDA: Training VO of EGEE– UK National Grid Service
• Under preparation for– US Open Science Grid, Economy-Grid,
Swiss BioGrid, BioInfoGrid, Turkish Grid
• P-GRADE portal can be installed for any public/private EGEE or Globus Grid/VO
Please contact us!
RReferenceseferences
![Page 30: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/30.jpg)
30CEOS WGISS Meeting, May 8-12, 2006, Budapest
Main conclusion:Main conclusion:Easy-to-use, technology-neutral Easy-to-use, technology-neutral
Grid portal for e-ScientistsGrid portal for e-Scientists• The P-GRADE Portal hides differences of Grids
• Globus X – LCG2 – gLite Grid interoperability• Transparent switching between Grid technologies
• Graphical tools for application development, execution and monitoring
• Seq & parallel components can be integrated into large Grid applications
• Your code does not have to contain grid specific calls• Support for collaborative research
• Share workflows• Share legacy applications and jobs as application services
• Built by standard portlet API• customizable to specific application areas and user groups
(such as earth scientists…)
![Page 31: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/31.jpg)
31CEOS WGISS Meeting, May 8-12, 2006, Budapest
Further information…Further information…
– Visit the www.lpds.sztaki.hu/pgportal site(manuals, slide shows, installation procedure, etc.)
– Visit or request a training event!(see list of events on P-GRADE Portal homepage)
• Lectures, demos, hands-on tutorials, application development support
– Get an account for a portal service:• VOCE portal - SZTAKI• SEEGRID portal – SZTAKI• HUNGrid portal – SZTAKI• NGS portal – University of Westminster
– Install a portal for your community: If you are the administrator of a Grid/VO SZTAKI is pleased to help you install a portal for your community!
![Page 32: 1 CEOS WGISS Meeting, May 8-12, 2006, Budapest MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences](https://reader035.vdocument.in/reader035/viewer/2022062618/5514e181550346935c8b566d/html5/thumbnails/32.jpg)
32CEOS WGISS Meeting, May 8-12, 2006, Budapest
• P-GRADE Portal tutorial in Paris (free event!)22-23, June – lectures, hands-on, application
supportSee details on P-GRADE Portal homepage!
• Joint Regional CE EGEE and SEE-GRIDSummer School on Grid Application Support
3-8 July, Budapest, SZTAKI
www.egee.hu/grid06
Thank you!Thank you!Questions?Questions?