p-grade portal family: past, present, future
DESCRIPTION
P-GRADE Portal Family: Past, Present, Future. Peter Kacsuk MTA SZTAKI Univ. of Westminster. www.lpds.sztaki.hu/pgportal pgportal @ lpds.sztaki.hu. SZTAKI Lab members. The community aspects of e-science. Web2 is about creating and supporting web communities - PowerPoint PPT PresentationTRANSCRIPT
1
www.lpds.sztaki.hu/[email protected]
P-GRADE Portal Family: Past, Present, Future
Peter KacsukPeter Kacsuk MTA SZTAKI
Univ. of Westminster
2
SZTAKI Lab members
3
The community aspects of e-science
• Web2 is about creating and supporting web communities• Grid is about creating virtual organizations where e-science
communities – can share resources and – can collaborate
• An e-science portal should support e-science communities in their collaborations and resource sharing
• And even more: it should provide simultaneous access to any accessible – Resources– Databases– Legacy applications– Workflows, etc.no matter in which grid they are operated on.
4
Who are the members of an e-science community?
End-users (e-scientists)• Execute the published applications with custom input parameters by creating application instances using the published applications as templates
Grid Application Developers• Develop grid applications by the portal• Publish the completed applications for end-users
5
What does an e-science community need?
App. Repository
Portal
Clouds
Local clusters
Supercomputers
Desktop grids (DGs)(BOINC, Condor, etc.)
Cluster based service grids (SGs)(EGEE, OSG, etc.)
Supercomputer based SGs
(DEISA, TeraGrid)
Grid systems
Application developers
E-scientists
6
Requirements for an e-science portal from the e-scientists’ point of view
It should be able to• Support large number of e-scientists (~ 100) with good
response time• Enable the store and share of ready-to-run applications• Enable to parameterize and run applications• Enable to observe and control application execution• Provide reliable appl. execution service even on top of
unreliable infrastructures (like for example grids)• Provide specific, user community views• Enable the access of the various components of an e-
science infrastructure (grids, databases, clouds, local clusters, etc.)
• Support user’s collaboration via sharing:– Applications (legacy, workflow, etc.)– Databases
7
Requirements for an e-science portal from the app. developers’ point of view
Beyond the the end-user requirements it should be able to– Enable the store and share of half-made
applications, application templates– Provide graphical appl. developing tools (e.g.
workflow editor) to develop new applications– Provide methods and API to customize the
portal interface towards specific user community needs by creating user-specific portlets
– Enable the integration/call of other services
8
Collaboration between application developers
App. Repository
Portal
E-science infrastructure
Application developers
• Application developers use the portal to develop complex applications (e.g. parameter sweep workflow) for the e-science infrastructure
• Publish templates, legacy code appls. and half-made applications in the repository to be continued by other appl. developers
9
Collaboration between e-scientists and application developers
App. Repository
Portal
Application developers
E-scientists
End-users (e-scientists)• Specify the problem/application needs• Execute the published applications via the portal with custom input parameters by creating application instances
Application Developers• Develop e-science applications via the portal in collaboration with e-scientists• Publish the completed applications for end-users via an application repository
10
The role of the Grid portal community
Grid Portal Developers• Jointly develop the portal core services (e. g. GridSphere, OGCE, Jetspeed-2, Liferay, etc.) • Jointly develop higher level portal services (workflow management, data management, etc.)• Jointly develop specialized/customized portal services (grid testing, rendering, etc.)• Use the power of the community to create really good portals
Grid Portal Administrators• Install the portal• Maintain and operate the portal• Give feedback for the portal developers
To organize this community we • Established the P-GRADE Portal Developer Alliance• Use sourceforge• Organized PUCOWO
11
P-GRADE portal family
• The goal of the P-GRADE portal family – To meet all the requirements of end-users and
application developers listed above– To provide a generic portal that can be used by
a large set of e-science communities– To provide a community code based on which
the portal developers’ community can start to develop specialized and customized portals (science gateways)
12
P-GRADE portal family
P-GRADE portal2.4
NGS P-GRADE portal
P-GRADE portal2.5
Param. Sweep
P-GRADE portal2.8
P-GRADE portal2.9.1
Current release
WS-PGRADE Portal
Beta release 3.1
WS-PGRADE Portal
Release 3.2
GEMLCAGrid Legacy Code Arch.
GEMLCA, repository concept
Basic concept
Open source from Jan. 2008
2008
2009
2010
2003
2006
13
Main features of P-GRADE portal
Supports • generic, workflow-oriented applications• parameter sweep (PS) applications with new super-
workflow concept– A. Balasko: Flexible PS application management in P-GRADE portal
• 3-level parallelism (MPI, WF-branch, PS)• Simultaneous access of wide variety of resources
– Z. Farkas: PBS and ARC integration to P-GRADE portal – P. Kacsuk: P-GRADE and WS-PGRADE portals supporting desktop
grids and clouds • Access to workflow repository
– Akos Balasko and Miklos Kozlovszky: SEE-GRID and EGEE Portal applications
• Development of application specific portals– Andreas Quandt and Lucia Espona Pernas: Portal for Proteomics – Tamas Kiss, Gabor Terstyanszky, Zsolt Lichtenberger, Christopher
Reynolds: Rendering Portal Service for the Blender User Community
14
Creating application specific portals from the generic P-GRADE portal
• Creating an appl. spec. portal does not mean to develop it from scratch
• P-GRADE is a generic portal that can quickly and easily be customized to any application type
• Advantage: – You do not have to develop the generic parts (WF
editor, WF manager, job submission, monitoring, etc.)– You can concentrate on the appl. spec. part– Much shorter development time
• See details in:– A. Balasko: Developing application specific portlets for
P-GRADE portal
15
Application Specific P-GRADE portals
Rendering portal by Univ. of Westminster
OMNeT++ portal by SZTAKI
Traffic simulation portal by Univ. of Westminster
16
Grid interoperation by P-GRADE portal
• P-GRADE Portal enables: Simultaneous usage of several production Grids at workflow level
• Currently connectable grids: – LCG-2 and gLite: EGEE, SEE-GRID, BalticGrid– GT-2: UK NGS, US OSG, US Teragrid– Campus Grids with PBS or LSF– BOINC desktop Grids– ARC: NorduGrid
• In prototype: – Clouds (Eucalyptus, Amazon)
• Planned: – UniCore: D-Grid (joint work with MosGrid)
17
User
P-GRADE Portal
SZTAKI Portal Server
Simultaneous use of production Grids at workflow level
WMS
broker
Workflow
Manchester
Leeds
UK NGS GT2
EGEE-VOCE gLite
Job
Job
Job
Job
Budapest
Athens
Brno
Supports both direct and brokered job submission
18
P-GRADE Portal references
• P-GRADE Portal services:– SEE-GRID, BalticGrid– Central European VO of EGEE– GILDA: Training VO of EGEE– Many national Grids (UK, Ireland, Croatia,
Turkey, Spain, Belgium, Malaysia, Kazakhstan, Switzerland, Australia, etc.)
– US Open Science Grid, TeraGrid– Economy-Grid, Swiss BioGrid, Bio and
Biomed EGEE VOs, MathGrid, etc.– EDGeS P-GRADE portal service
19
Community based business model for the sustainability of P-GRADE portal
• New features that can be written in a proposal are developed in EU projects. Examples:– PS feature: SEE-GRID-2
– Integration with DSpace: SEE-GRID-SCI
– Integration with BOINC: EDGeS, CancerGrid
– Supporting workflow interoperability: SHIWA
– Further extensions: new INFRA-2011-1.2.1: e-Science environments project
• We use internal budget from the Hungarian Academy – for features that are not financed by projects
– for bug fixing, maintenance, etc.
20
Community based business model for the sustainability of P-GRADE portal
• There is an open Portal Developer Alliance • Objective:
– to develop a more and more usable portal for the e-science community
– Anything that is developed by this community goes to the open source version of P-GRADE portal at sourceforge under GPL license
• Current active members:– Middle East Technical Univ. (Ankara, Turkey)
• gLite file catalog management portlet– Univ. of Westminster (London, UK)
• Developed the UK NGS P-GRADE portal member of the P-GRADE portal family
– ETH Zürich• ARC integration• PBS integration
– You are kindly invited to join the alliance
21
Business model for the sustainability of P-GRADE portal
• Some of the developments are ordered by customer academic institutes:– Collaborative WF editor: Reading Univ. (UK)– Accounting portlet and SLA support: MIMOS (Malaysia)
• Mohd Sidek Salleh and Z. Farkas: P-GRADE portal developments in the framework of the MIMOS-SZTAKI joint project
– Separation of front-end and back-end: MIMOS• Mohd Sidek Salleh: Usability and Performance Improvements of P-GRADE
portal in KnowledgeGRID Malaysia – Shiboleth integration: ETH Zurich– PBS, LSF, ARC integration: ETH Zurich
• Benefits for the customer academic institutes: – Basically they like the portal but they have some special needs that
require extra development– Instead of developing from scratch a new portal (using many person-
months) rather they pay only for the required little extension/modification of the portal
– To solve their problem gets priority in SZTAKI– They become expert of the internal structure of the portal and will be able
to further develop it according to their needs– Joint publications
22
Main features of NGS P-GRADE portal
• Advantages:– Extends P-GRADE portal with
• GEMLCA legacy code architecture and repository (nodes of a workflow can be taken from the GEMLCA repository)
• SRB file management• OGSA-DAI database access• WF level interoperation of grid data resources • Workflow interoperability support
– All these features are provided as production service for the UK NGS
– See the talk:• Tamas Kiss, Tamas Kukla, Gabor Terstyanszky: P-GRADE portal
extensions for UK NGS users
• Drawbacks– Based on P-GRADE version 2.4– There is no parameter sweep support
23
WS-PGRADE and gUSE
• New product in the P-GRADE portal family:– WS-PGRADE (Web Services Parallel Grid Runtime and Developer Environment)
• WS-PGRADE uses the high-level services of– gUSE (Grid User Support Environment) architecture
• Integrates and generalizes P-GRADE portal and NGS P-GRADE portal features– Advance data-flows (PS features)– Built-in GEMLCA– Built-in Workflow repository
• gUSE advanced features– Scalable architecture (written as set of services and can be installed on one or more
servers)– Can execute simultaneously very large number of jobs (100.000 – 1.000.000)– Various grid submission services (GT2, GT4, LCG-2, gLite, BOINC, local)– Built-in inter-grid broker (seamless access to various types of resources and grids)
• Comfort features– Different separated user views supported by gUSE application repository
• See details in:– M. Kozlovszky and Peter Kacsuk: WS-PGRADE portal and its usage in the
CancerGrid project – WS-P-GRADE portal tutorial
• Drawback:– Not as stable and matured as P-GRADE
24
Ergonomics
• Users can be grid application developers or end-users. • Application developers design sophisticated dataflow graphs
– embedding into any depth, conditional structures, generators and collectors at any position
– Publish applications in the repository at certain stages of work• Applications• Projects• Concrete workflows• Templates• Graphs
• End-users see WS-PGRADE portal as a science gateway – List of ready-to-use applications in gUSE repository– Import, parameterize and execute applications without knowledge of
programming, dataflow or grid
25
Current users of WS-PGRADE• CancerGrid project
– Predicting various properties of molecules to find anti-cancer leads
– Creating science gateway for chemists• EDGeS project (Enabling Desktop Grids for
e-Science)– Integrating EGEE with BOINC and
XtremWeb technologies– User interfaces and tools
• ProSim project– In silico simulation of intermolecular
recognition – JISC ENGAGE program (UK)
• Public WS-PGRADE portal servicehttps://guse.sztaki.hu/gridsphere/gridsphere
– We encourage you to play with it to get feedback from you concerning features and stability
– You can also get the install package
26
P-GRADE portal family summary
P-GRADE NGS P-GRADE WS-PGRADE
Scalability ++ + +++
Repository DSpace/WF Job & legacy code services
Built-in WF repository
Graphical workflow editor
+ + +
Parameter sweep support
+ - ++
Access to various grids
GT2, LCG-2, gLite, BOINC, ARC, campus
GT2, LCG-2, gLite, GT4
GT2, LCG-2, gLite, GT4,
BOINC, campus
Access to clouds In prototype - In progress
Access to databases
- via OGSA DAI SQL
Support for WF interoperability
Planned in SHIWA
+ Planned in SHIWA
27
Useful information• P-GRADE:http://portal.p-grade.hu/
– User Manuals– Admin Manuals– Technical Documentation– Tutorials
• P-GRADE at sourceforge:http://sourceforge.net/projects/pgportal/• P-GRADE at wikipedia:http://en.wikipedia.org/wiki/P-GRADE_Portal• 3G Bridge at sourceforge:http://sourceforge.net/projects/edges-3g-bridge/• WS-PGRADE/gUSE:http://www.guse.hu/• NGS P-GRADE:http://ngs-portal.cpc.wmin.ac.uk/index.php/Main_Page
28
Requests• We collect your feature requests to improve the portals• We also have 3 requests for you:
1. We would like to collect publications related with the portals in a repository, so please, send us your existing and future publications
2. In your publications please reference the following publications on the portals:
– P. Kacsuk and G. Sipos: Multi-Grid, Multi-User Workflows in the P-GRADE Portal, Journal of Grid Computing, Vol. 3, No. 3-4, Springer, pp. 221-238, 2005
– P. Kacsuk, T. Kiss, G. Sipos: Solving the Grid Interoperability Problem by P-GRADE Portal at Workflow Level, Future Generation Computing Systems, Vol. 24, Issue 7, pp. 744–751, 2008
3. Please, show your appreciation at sourceforge Ratings and Reviews:http://sourceforge.net/projects/pgportal/
Please, use the sourceforge discussion forum and feature request at:http://sourceforge.net/projects/pgportal/forums/forum/769302http://sourceforge.net/tracker/?group_id=213502&atid=1025973
29
Further plans• Moving from Gridsphere to Liferay (close to prototype level in WS-
PGRADE)• Creating the production version of cloud integration• To make WS-PGRADE open source• SHIWA - incorporating other workflow systems:
– Triana– ASKALON– Taverna– Pegasus– Kepler, etc.
• Connecting to Unicore with MosGrid• Taking into account requirements of PUCOWO• Submitting an FP7 project proposal for the
INFRA-2011-1.2.1: e-Science environments FP7 CALL• Next event possibility:
– PARENG’2011: The Second International Conference onParallel, Distributed, Grid and Cloud Computing for Engineering, Ajaccio, Corsica, France, 12-15 April 2011
– I organize a special session on Science Gateways for Grid and Cloud Systems
– http://www.civil-comp.com/conf/pareng2011.htm#sessions
30
Thank you for your attention!Any questions?
www.portal.p-grade.hu www.wspgrade.hu