infso-ri-508833 enabling grids for e-science supporting legacy code applications on egee vos by...
Post on 02-Jan-2016
216 Views
Preview:
TRANSCRIPT
INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
Supporting legacy code applications on EGEE VOs by GEMLCA and the P-GRADE portal
P. Kacsuk*, G. Sipos* and T. Kiss***MTA SZTAKI** Univ. of Westminster
Enabling Grids for E-sciencE
INFSO-RI-508833
Properties of the P-GRADE Portal
• What is P-GRADE portal?– General purpose, graphical, workflow-oriented Grid portal– Supports the development and execution of workflow-based Grid
applications– Components of the workflows can be
Sequential jobs Parallel jobs (MPI, PVM) Legacy code (GEMLCA) services
– Easy porting of legacy and other applications to the Grid– Enables the exploitation of two levels of parallelism
intra-job inter-job
– Enables to control and observe the execution of the workflow applications
– Enables fault-tolerant workflow execution
Enabling Grids for E-sciencE
INFSO-RI-508833
Properties of the P-GRADE Portal
• Enables application-specific extensions by new portlets (E-GRID, Traffic simulation Grid, etc.)
• Hides the low-level Grid details • Enables the access of every important feature of the
underlying Grid, e.g.: – brokering service– information service, etc.
• Can serve as a bridge between different Grids to solve the Grid interoperability problem at the workflow level:– Enables easy porting of legacy and other applications
between various Grids (e.g. from LCG to gLite, from GT2 to GT4, etc.)
– Hides middleware changes for the end-user– Enables parallel execution of a workflow in several Grids
Enabling Grids for E-sciencE
INFSO-RI-508833
Properties of the P-GRADE Portal
• Enables application-specific extensions by new portlets (E-GRID, Traffic simulation Grid, etc.)
• Hides the low-level Grid details • Enables the access of every important feature of the
underlying Grid, e.g.: – brokering service– information service, etc.
• Can serve as a bridge between different Grids to solve the Grid interoperability problem at the workflow level:– Enables easy porting of legacy and other applications
between various Grids (e.g. from LCG to gLite, from GT2 to GT4, etc.)
– Hides middleware changes for the end-user– Enables parallel execution of a workflow in several Grids
Enabling Grids for E-sciencE
INFSO-RI-508833
Proxy management
P-GRADE portal in a nutshell
Grid resources management
Workflow creation
Job mapping to Grid resources
Workflow management and
execution visualization
Enabling Grids for E-sciencE
INFSO-RI-508833
Two levels of parallelism by the P-GRADE workflow
• Semantics of the workflow enables two levels of parallelism:
The job can be a parallel program
– Parallel execution inside a workflow node– Parallel execution among workflow nodes
Multiple jobs can run parallel
• The P-GRADE Portal workflow concept enables the efficient parallelization of complex problems
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Grid Execution Management for Legacy
Code Applications
OGSA-DAIIntegration with P-GRADE
Tamas Kukla and Tamas KissCentre for Parallel Computing
University of WestminsterT.Kukla@student.westminster.ac.uk, kisst@wmin.ac.uk
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
OGSA-DAI portlets
4 portlets implemented
1. Data Service Manager
2. Database Browser
3. Query Manager
4. Data Manipulation
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Data Service Manager portlet
Add/remove OGSA-DAI services
List available resources
of a service
List available
tables
Select and describe
tables
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Database Browser portlet
Select service /
resource / table
Display the
selected table or XML
resource
Order by the
selected column
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Query Manager portletDisplay results on screen
Run query on selected database
See next slide
Display results
Display results
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Query Manager portletDeliver results to file
Data can be sliced up
and compressed
before delivery
Define target
location
Define query
statement
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Data Manipulation portletManipulate data of selected database
Select database
Define update
statement
See next slide
Update
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Data Manipulation portletDeliver results from file to database
Define target table
Define files to be delivered
Extracts and
merges
Select target
service / resource
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
A possible user scenario
• The new OGSA-DAI portlet allows to browse an OGSA-DAI database and deliver query results to a set of files.
• These files can be used in normal or parameter study workflows as input ports.
• The workflow output can be converted and transferred back to the database.
1. Select the data service and resource
2. Run a query and deliver results to a (set of) file(s)
3. Run the (parameter study) workflow on the data-sets
4. Transfer the results back into an OGSA-DAI database
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Extended PG-Portal
DataBrowserWorkflow
Step1: Select the OGSA-DAI service and DataBase resource with the DataBrowser
OGSA-DAIservice
Oracle DataBaseresource
User
eXist XMLDataBaseresource
RequestList of resources
Request
List of resources
Storage resource (local or remote)
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Step1: Select the OGSA-DAI service and DataBase resource with the DataBrowser
Select service and resource
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Extended PG-Portal
DataBrowserWorkflow
Step2: Run a query and deliver results to a (set of) file(s)
OGSA-DAIservice
Oracle DataBaseresource
User
eXist XMLDataBaseresource
Query Results
Query
ResultsResults
Query
Storage resource (local or remote)
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Step2: Run a query and deliver results to a (set of) file(s)
Define query
statement
Deliver results to a set of files
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Extended PG-Portal
DataBrowserWorkflow
Step3: Run (parameter study) workflow (this step does not use the OGSA-DAI portlets!)
OGSA-DAIservice
Oracle DataBaseresource
User
eXist XMLDataBaseresource
Create workflow Workflow
output
Storage resource (local or remote)
Input data
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Extended PG-Portal
DataBrowserWorkflow
Step4: Transfer results back into an OGSA-DAI database
OGSA-DAIservice
Oracle DataBaseresource
User
eXist XMLDataBaseresource
Workflow output
Storage resource (local or remote)
Workflow output
Workflow output
Confirmation
Confirmation
Confirmation Request
www.cpc.wmin.ac.uk/GEMLCAwww.cpc.wmin.ac.uk/GEMLCA
Step4: Transfer results back into an OGSA-DAI database
Define target
databaseDefine target table
Define files to be transferred
and converted
top related