er-flow c. vuerli contributions by g. terstyanszky, k. varga
TRANSCRIPT
ER-flow
C. Vuerli
Contributions by G. Terstyanszky, K. Varga
University of Westminster UoW United KingdomMagyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete
MTA-SZTAKI Hungary
Centre National de la Recherche Scientifique CNRS FranceStichting European Grid Initiative EGI.eu The NetherlandsAcademic Medical Center of the University of Amsterdam
AMC The Netherlands
Technische Universität Dresden TUD GermanyLudwig-Maximilians-Universität München LMU GermanyUniversity College London UCL United KIngdomTrinity College Daudublin TCD IrelandIstituto Nazionale di Astrofisica INAF Italy
Partners:
Technology providers:CNRS, EGI.eu, MTA-SZTAKI, UoW
Research Communities:Astro-Physics INAFComputational Chemistry LMU + TUDHelio-Physics TCD + UCLLife Science AMC
Duration:September 2012 – August 2014
ER-flow Project
SHIWA vs. ER-flow
• What are we going to use?– SHIWA technology– ER-flow dissemination, user support
• How and where should we use the SHIWA and the ER-flow name?– On leaflets, posters, presentations?
Project Aim and ServicesAim:• To provide a simulation platform for research communities to enable
seamless execution of workflows of different workflow systems through workflow interoperability
• To investigate data interoperability issues in the workflow domain and propose solutions
Services:• To support the whole workflow lifecycle: editing, uploading, browsing
downloading and executing workflows • To provide coarse-grained workflow interoperability solution • To provide GUIs to manage workflows
Key actors:• Researchers• workflow engine developers• workflow developers
Objective No. 1: To further build a European community of workflow developers and users involving a wide range of research communities which already use workflow systems and which are new to this technology.
Objective No. 2: To migrate workflow based scientific applications of the supported research communities to the European Grid Infrastructure through the SHIWA Simulation Platform and to use these applications both for production runs and to promote e-Science workflow solutions for research communities.
Objective No. 3: To disseminate the workflow interoperability solution of the SHIWA project among the selected research communities and identify further research communities that need the simulation platform to run their experiments.
Objective No. 4: To define requirements of the supported research communities on interoperability of the scientific data in the workflow domain and to identify existing and missing protocols and standards needed to support this interoperability.
Objective No. 5: To write a study on the interoperability of the scientific data in the workflow domain, make recommendations on how to achieve data and workflow interoperability with existing protocols and standard, and identify research, development and standardisation issues required to be solved in order to achieve workflow interoperability in data-intensive research.
Project Objectives
No. description M12 M24 milestones WP
01 new research communities of SSP
2 4 M2.2/M2.3 WP2
02 number of researchers using SSP
100 250 M5.2/M5.3 WP5
Objective No. 1: To further build a European community of workflow developers and users involving a wide range of research communities which already use workflow systems and which are new to this technology.
Objectives and Indicators
Questions:- Which communities are we addressing? - How will we reach 250 users? Which communities will they represent?- Will they workflow developers or researchers with basic IT knowledge?
Objective No. 2: To migrate workflow based scientific applications of the supported research communities to the European Grid Infrastructure through the SHIWA Simulation Platform and to use these applications both for production runs and to promote e-Science workflow solutions for research communities.
No. description M12 M24 milestones WP
03 Astro-Physics workflows 6 14 M5.2/M5.3 WP5
04 Computational Chemistry workflows
10 20 M5.2/M5.3 WP5
05 Helio-Physics workflows 6 14 M5.2/M5.3 WP5
06 Life Science workflows 10 20 M5.2/M5.3 WP5
07 number of executed workflows
1000 3000 M5.2/M5.3 WP5
Objectives and Indicators
Questions:- Do we have these workflows? If we don’t have them how will we identify
and find them? Will they non-native or meta-workflows?- Do we have “real” users who will run 3000 times workflows available in
the repository?
Objective No. 3: To disseminate the workflow interoperability solution of the SHIWA project among the selected research communities and identify further research communities that need the simulation platform to run their experiments.
No. description M12 M24 milestones WP
08 dissemination & training events for supported
communities
4 8 M2.2/M2.3 WP2
09 dissemination & training events for supported
communities
2 4 M2.2/M2.3 WP2
10 European dissemination & training events
2 4 M2.2/M2.3 WP2
11 users of the User Forum 200 400 M2.2/M2.3 WP2
12 publications on research achievements
4 8 M5.2/M5.3 WP5
Objectives and Indicators
Questions:- What kind of dissemination and training events will we run? How will
identify the dissemination and training events? Who will run these events?
- How will we run the User Forum? How will we attract users to the forum?
Objective No. 4: To define requirements of the supported research communities on interoperability of the scientific data in the workflow domain and to identify existing and missing protocols and standards needed to support this interoperability.
No. description M12 M24 milestones WP
13 requirements specification for data interoperability
2 2 M3.1/M5.1 WP2WP5
Objectives and Indicators
Questions:- What kind of data interoperability issues will we target?- How will we identify these requirements? Who identify these
requirements?
Objective No. 5: To write a study on the interoperability of the scientific data in the workflow domain, make recommendations on how to achieve data and workflow interoperability with existing protocols and standard, and identify research, development and standardisation issues required to be solved in order to achieve workflow interoperability in data-intensive research.
No. description M12 M24 milestones WP
14 recommendations on data interoperability
0 1 M4.2 WP4
15 publications on data interoperability
0 4 M4.2 WP4
Objectives and Indicators
Work Packages and Efforts
No. title m/m WP leader
starting ending
WP1 Project Management 6 UoW 01 24
WP2 Knowledge Transfer 29 SZTAKI 01 24
WP3 Infrastructure & Technical Support
20 UoW 01 24
WP4 Data Interoperability in Workflow Domain
30 CNRS 01 24
WP5 Application Support 76 AMC 01 24
partners wp1 wp2 wp3 wp4 wp5
UoW 6 2 11 2 3
MTA-SZTAKI 8 4 2 4
CNRS 1 5 15 1
EGI.eu 6 2 4
AMC 3 3 19
partners wp1 wp2 wp3 wp4 wp5
TUD 2 9
LMU 4 8
UCL 2 1 5
TCD 1 8
INAF 3 2 15
Partners and Efforts
NA2 Tasks• Task 2.1: Maintaining the SHIWA website and
producing dissemination material (M01-M24)• Task 2.2: Running the SHIWA User Forum (M01-M24)• Task 2.3: Organising dissemination events for
European research communities (M01-M24)– One dissemination event for each supported research
community in each project year– At least two events for new research groups in the
second project year• Task 2.4: Running training courses (M06-M24)
– Organising one training course each project year
NA2 Strategy• Organize user-community-targeted events/workshops• Organize user-community-targeted training events• Attend and organize scientific / non-scientific events• Provide Web visibility (User Forum, ER-flow website,
SHIWA website, SHIWA Facebook)– Improve the website to be more informative
feedback required• Initiate collaborations with other projects (MoU)
– SCI-BUS and EGI InSPIRE first
abstract workflow
concrete workflow
workflow engine
workflow configuration
concrete workflow
engine
abstract workflow
engine
workflow engine configuration
workflow formal descriptionworkflow engine formal description
CGI Concept
WF Submission Service
Workflow Engine B
Workflow of Workflow Engine B
Workflow Engine A
CGI: submitting non-native workflow
DCI
• non-native workflow: WF• non-native workflows are black boxes which are managed
as legacy code applications
Workflow Engine A
CGI: meta-workflow submission
• native workflows: J1, J3 and WF2• non-native workflows: WF4
• black boxes which are managed as legacy code applications
WF2 WF3
J1
WF4
Submission Service
Workflow Engine B
Workflow of Workflow Engine B
Workflows of Workflow Engine A
DCI
Workflow Engine A
Workflow Repository
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
SHIWA Portal
GEMLCA admin
SHIWA Science Gateway
WE1 WEp
GEMLCA Repository
WF1 WFm
SHIWA Repository
WF1 WFn
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
GEMLCA Service
GEMLCA with GIB
SHIWA Proxy Server
Proxy Server
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
SHIWA Science Gateway Resourcesnative WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.5 WE submission to local cluster repository GEMLCA + SHIWA repo remote resources: through remotely pre-deployedsubmitter GEMLCA with GIB WEs to ARC, gLite, Globus Unicore DCIsproxy management SHIWA Proxy Server
SSP: CGI Infrastructure
CGI Developer Scenario: Specifying Workflow Engine
SHIWA Portal
GEMLCA admin
SHIWA Science Gateway
WE1 WEp
GEMLCA Repository
WF1 WFm
SHIWA Repository
WF1 WFn
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
GEMLCA Service
GEMLCA with GIB
SHIWA Proxy Server
Proxy Server
step 1specify WE
data
step 2upload WE
binary, dependencies
step 3deploy WE
workflow engine
developer
CGI Developer Scenario: Specifying Non-Native Workflows
SHIWA Portal
SHIWA Science Gateway
WE1 WEp
GEMLCA Repository
WF1 WFm
SHIWA Repository
WF1 WFn
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
GEMLCA Service
GEMLCA with GIB
SHIWA Proxy Server
Proxy Server
step 1specify WF
data
step 2upload WF
step 3deploy WF
workflow developer
CGI User Scenario: PGRADE as Native WE
SHIWA Repository
SHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WF1 WFm
GEMLCA with GIB
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
WF liste-scientists
SHIWA Proxy Server
Proxy Server
step 1search WF
step 2 edit WF
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
step 7 run WFstep 6
retrieve proxy
WE + WF step 5retrieve WF
step 4submit WF
step 3retrieve WF data
SHIWA Science Gateway
GEMLCA Service
SHIWA Repository
Submission service
WE1 WEn
WF1 WFm
user
SHIWA Proxy Server
Proxy Server
step 1search WF
step 2 edit WF
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
step 6 run WF
step 5retrieve proxy
WE + WF
step 4retrieve WF
step 3submit WF
GEMLCA Client
MOTEURWorkflow
Engine
MOTEUR Workflow
editor
GEMLCA UI
CGI User Scenario: MOTEUR as Native WE
SSP: Execution Infrastructure
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
Galaxy WE
Pegasus WE
ProActiveWE
Workflow Engines
Astrophysics Science Gateway
Computational Chemistry Science Gateway
Heliophysics Science Gateway
Life sciences Science Gateway
SHIWA RepositoryWFn
SHIWA Submission
Service
WFn
submitting native
workflows
submitting non-native workflows
Export to IWIR
Import from IWIR
WFBWFA
Interoperable Workflow Intermediate Representation IWIR
Fine-grained interoperability (FGI)
Robot Certificate Management
Workflow Execution Statistics
User Communities and Simulation Platform
Research communities supported by the project• Astro-Physics• Computational Chemistry• HelioPhysics• Life Science
Further research communities• at least four further research communities will be supported
candidate communities• Hydrometeorology• Seizmology• Further communities considered