gsoc airavata
DESCRIPTION
TRANSCRIPT
![Page 1: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/1.jpg)
Apache Airavata
GSOC 2013
![Page 2: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/2.jpg)
Knowledge and Expertise
Computational Resources
Scientific Instruments
Algorithms and Models
Archived Data and Metadata
Advanced Science Tools
Target Community: Science Gateways Enabling & Democratizing Scientific Research
![Page 3: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/3.jpg)
What does Apache Airavata do?• Compose, manage, execute,
and monitor distributed, computational workflows.
• Wrap legacy command line scientific applications with Web services.
• Run jobs on computational resources ranging from local resources to computational grids and clouds.
• Manage provenance data.
![Page 4: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/4.jpg)
Workflow Interpreter
Application Factory
Message Box
Registry
Apache Airavata
API
Lorem ipsum
insolens
p1m5
duo x
End
Use
rsG
atew
ay
Dev
elop
er
Scientific Applicati
on
Core Developer
Computational Resources
Apache Airavata
![Page 5: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/5.jpg)
Apache Airavata ComponentsComponent Description
XBaya Workflow graphical composition tool.
Registry Service Insert and access application, host machine, workflow, and provenance data.
Workflow Interpreter Service
Execute the workflow on one or more resources.
Application Factory Service (GFAC)
Manages the execution and management of an application in a workflow
Messaging System WS-Notification and WS-Eventing compliant publish/subscribe messaging system for workflow events
Airavata API Single wrapping client to provide higher level programming interfaces.
![Page 6: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/6.jpg)
Hi, I’m Nolram. I’m a computational
physicist. I run computational
experiments everyday
This is how typically I run my experiments
![Page 7: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/7.jpg)
Scientific Application
Another ScientificApplication
First I collect my observed data
And then pass data to my applications & get
the result
This is starting to become a very tiring
task
![Page 8: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/8.jpg)
How can I make this much simpler…?
Logically, this is how my life would be
made easier…
Is it possible to automate this flow
sequence without my guidance?
![Page 9: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/9.jpg)
Scientists from many different fields face this
problem everyday.
The solution is to use a workflow-powered science gateway to
manage the experiment online.
What is a workflow you ask? Well, you just saw one in
our previous animation…
![Page 10: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/10.jpg)
We introduce Apache Airavata, a system capable of composing, managing, executing, and monitoring
small to large scale applications and workflows
Want to see how it works?
A Typical Workflow
![Page 11: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/11.jpg)
Apache Airavata
I will handover my data & my experiment details (the workflow)
to the Airavata server
The Gateway
Airavata will complete the experiment & return me the results
Results
Progress of the experiment
… and while I wait for results, Airavata will notify me with
progress updates of my experiment
![Page 12: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/12.jpg)
Let’s look closely how Airavata manages workflows.
The GatewayResults
Experiment progress
Apache Airavata
![Page 13: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/13.jpg)
Let’s look closely how Airavata manages workflows.
The GatewayResults
Experiment progress
![Page 14: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/14.jpg)
Airavata main has 4 components…
The Gateway
1. Workflow InterpreterSteer the workflow execution
2. The GFacSteer science app executions & data
transfers
Workflow Interpreter
GFac
Message Box
Registry
3. The RegistryDefines the available applications &
records all results of experiments
4. The Message BoxRecords the progress of the workflow
execution
![Page 15: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/15.jpg)
A Stable API for Airavata
Apache Airavata
Lorem ipsum
insolens
p1
m5
duo x
End
Use
rsG
atew
ay D
evel
oper
Scientific Application
Computational Resources
![Page 16: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/16.jpg)
Application Registration
UIApplication Developer
A1
Airavata Service Interface
(wraps client API)
Service Map XML
Airavata Server
Service Map to AWSDL
Web Based workflow composerWorkflow
Developer
Get AWSDL
Put XWF
A2
A3
W1W2
W3
Web Based Experiment
BuilderExperimentBuilder
Get WI’s
E1
E2
E3
W4
Shred Workflow
Inputs
Launch Workflow
Web Based Workflow MonitorWatch Progress
M1M2
Get Workflow Graph
M3 Monitor Workflow
![Page 17: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/17.jpg)
Goal of the project
• Design Web-Based interfaces for Airavata:– Application Registration– Workflow Construction– Workflow Execution– Workflow Monitoring
• Provide an opportunity for GSoC to understand Distributed System in action
• Scope for Research and Software Engineering papers
![Page 18: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/18.jpg)
Data Model
• Application Description – User describes inputs and outputs of the
application. – Currently this information is captured in Service
Map Schema. – This schema is stored in Airavata Registry as XML.
Also the schema utility generates a application service WSDL from this schema using the Airavata WSDL Generator.
![Page 19: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/19.jpg)
Application Registration
UI
Application Developer
A1
Service Map XML
Service Map to AWSDL
Web Based workflow composer
WorkflowDeveloper
Get AWSDL
A2
W1
W2
XML
Aira
vata
Ser
ver A
PI
Launch & Manage Jobs
Notify progress of job or workflow execution
Real-Time Monitoring
Messaging Subsystem
Registry
Execute & Manage Computations
Workflow Interpreter
Application Factory (Gfac)
Application Desc
Application Desc
Workflow
![Page 20: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/20.jpg)
A peek at one of the cluster
InterconnectNodes
![Page 21: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/21.jpg)
Scheduling ‘qsub’ batch jobs on the cluster worker node
worker node
worker node
worker node
worker node
SGE MASTER node
Queue-A Queue-B Queue-C
A S
lot 1
A S
lot 2
B S
lot 1
C S
lot 1
C S
lot 2
C S
lot 3
B S
lot 1
B S
lot 2
B S
lot 3
B S
lot 1
C S
lot 1
C S
lot 2
A S
lot 1
B S
lot 1
C S
lot 1
Queues
Policies
Priorities
Share/Tickets
Resources
Users/Projects
JOB Y JOB ZJOB X
JOB U
JOB OJOB N
![Page 22: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/22.jpg)
System characteristics
System statusResources
Job policiesResources
Resource Matching
JOB
User
User policiesGroupsRolesDepartmentsProjects
Selection Scheduling
![Page 23: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/23.jpg)
Simplified Gateway Architecture
One time Gateway Community Setup
Community Account Grid Certificate username, password
Gateway InterfaceGateway Server
Compute Servers
Gateway Authentication
Fetch
Community C
reden
tialGrid
Prox
y
Job Submit or File Transfer request
Output
Proxy, Job Request
Job Status, Output
Step 0
Step 1
Step 2,3,,
XSEDE User Portal
![Page 24: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/24.jpg)
ParamChem
BioVLab
GridChem
DES
VLAB
UltraScan
Apache Airavata 1.0
NSG
ParamChem
BioVLab
CIPRES
POPLAR
GridChem
DES UltraScan
VLAB
…
…
…
Apache Airavata 2.0
Apache Airavata 1.0
Apache Airavata 1.0
Apache Airavata 1.0
Apache Airavata 1.0
Apache Airavata 1.0
![Page 25: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/25.jpg)
![Page 26: Gsoc airavata](https://reader033.vdocument.in/reader033/viewer/2022061210/548f3771b47959d9168b4943/html5/thumbnails/26.jpg)