indigo paas tutorial - istituto nazionale di fisica...
TRANSCRIPT
![Page 2: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/2.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
INDIGOPAASTutorial
▪ IntroductoryConcepts• TOSCA
• Ansible
• Docker
• OrchestratorAPIs
• INDIGOTOSCAcustomtypesandtemplates
▪Demos
2
![Page 3: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/3.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
TOSCA
▪TopologyandOrchestrationSpecificationforCloudApplications▪Standardizesthelanguagetodescribe• ThestructureofanITService(itstopologymodel)
• Howtoorchestrateoperationalbehavior(planssuchasbuild,deploy,patch,shutdown,etc.)
- LeveragingtheBPMNstandard
• Declarativemodelthatspansapplications,virtualandphysicalinfrastructure
3
![Page 4: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/4.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
TOSCAinanutshell
4
![Page 5: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/5.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
TOSCATopology
5
TOSCA-Simple-Profile-YAML-v1.0-csprd02 04 February 2016 Standards Track Work Product Copyright © OASIS Open 2016. All Rights Reserved. Page 19 of 267
will execute the script automatically to create the node on db_server, but first it will prepare the local environment with the declared inputs for the operation. In this case, the orchestrator would see that the db_data input is using the get_artifact function to retrieve the file (my_db_content.txt) which is associated with the db_content artifact name prior to executing the db_create.sh script. The logical diagram for this example would appear as follows:
Note that while it would be possible to define one node type and corresponding node templates that represent both the DBMS middleware and actual database content as one entity, TOSCA normative node types distinguish between middleware (container) and application (containee) node types. This allows on one hand to have better re-use of generic middleware node types without binding them to content running on top of them, and on the other hand this allows for better substitutability of, for example, middleware components like a DBMS during the deployment of TOSCA models.
2.5 TOSCA template for a two-tier application The definition of multi-tier applications in TOSCA is quite similar to the example shown in section 2.2, with the only difference that multiple software node stacks (i.e., node templates for middleware and application layer components), typically hosted on different servers, are defined and related to each other. The example below defines a web application stack hosted on the web_server “compute” resource, and a database software stack similar to the one shown earlier in section 6 hosted on the db_server compute resource.
Example 6 - Basic two-tier application (web application and database server tiers)
tosca_definitions_version: tosca_simple_yaml_1_0 description: Template for deploying a two-tier application servers on two
▪3layers•Infrastructure(CloudorDCobjects)
•PlatformorMiddleware (Appcontainers)
•Applicationmodules,schemasandconfigurations
!
▪Relationshipsbetweencomponents:
•What’shostedonwhatorinstalledonwhat
•What’sconnectedtowhat
![Page 6: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/6.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
TOSCATopology
▪ComponentsinthetopologyarecalledNodes
▪EachNodehasaType(e.g.Host,BD,Webserver).
•TheTypeisabstractandhenceportable•TheTypedefinesPropertiesandInterfaces▪AnInterfaceisasetofhooks(namedOperations)
▪NodesareconnectedtooneanotherusingRelationships▪BothNodeTypesandRelationshipTypescanbederived
6
![Page 7: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/7.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
NodeTypeExamples
▪DefinespropertiesasYAMLmaps
▪Mightdefinescapabilities(Whatitcanprovidetoothernodes)
7
![Page 8: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/8.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
NodeTypeExamples
▪MightDefineRequirements(whatitneedsfromothernodes)
8
![Page 9: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/9.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
NodeTemplate
▪Aninstanceofatype(likeObjecttoClass)▪Hasspecificproperties▪Hasartifacts:•Whattoinstall
• Howtoinstall(mappedtointerfacehooks)
▪Hasrequirementsandcapabilities(orrelationships)
9
![Page 10: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/10.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
TopologyTemplateExample
10
![Page 11: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/11.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
INDIGOcustomtypes
11
▪ExtendtheTOSCAnormativetypes
E.g.:tosca.nodes.indigo.GalaxyPortal,tosca.nodes.indigo.GalaxyWN
Ansible playbook
![Page 12: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/12.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
AutomatedDeploymentwithAnsible
▪Ansibleisaconfigurationmanagementandprovisioningtool,similartoChef,PuppetorSalt.
▪Agentless• usesSSHbydefaulttomaketheconnectiontothetargetnode
• canberunlocally
▪ConfigurationinYAML
▪Veryeasytolearn• http://docs.ansible.com/ansible/index.html
12
![Page 13: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/13.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
Buildingblocks:Playbook
▪Definessequencesoftasks(Plays)tobeexecutedonagroupofhosts
•describespoliciesmachinesundermanagementshallenforce
•containsvariables,tasks, handlers,files,templatesandroles
•expressedinYAML
•Idempotent
13
--- - hosts: webservers vars: http_port: 80 max_clients: 200 remote_user: root tasks: - name: ensure apache is at the latest version yum: name=httpd state=latest - name: write the apache config file template: src=/srv/httpd.j2 dest=/etc/httpd.conf notify: - restart apache - name: ensure apache is running (and enable it at boot) service: name=httpd state=started enabled=yes handlers: - name: restart apache service: name=httpd state=restarted
![Page 14: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/14.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
Buildingblocks:Roles
▪Thebestwaytoorganizeyourplaybooks.•structurecontentintorelatedvars,tasks,files,handlers,etc.•filestructureforautomatedinclusionofrolespecificcontent
•rolescanbesharedandpulledfromAnsibleGalaxy,GitHub,etc.
14
--- - hosts: webservers vars: http_port: 80 max_clients: 200 roles: - webserver
![Page 15: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/15.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
AnsibleGalaxy:indigo-dcroles
15
ansible-galaxy install indigo-dc.<role>
![Page 16: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/16.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
Docker:Build-Ship-Run
▪Dockerisanopen-sourceenginetoeasilycreatelightweight,portable,self-sufficientcontainersfromanyapplication.
▪Thesamecontainerthatadeveloperbuildsandtestonalaptopcanrunatscale,inproduction,onVMs,OpenStackcluster,publiccloudsandmore.
▪Dockerfeatures:• versioning(git-like)
• componentre-use
• sharing(publicrepository)
16
![Page 17: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/17.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
Dockerfile
▪ADockerfileisascript,composedofvariouscommands(instructions)andargumentslistedsuccessivelytoautomaticallyperformactionsonabaseimageinordertocreate(orform)anewone
17
FROM ubuntu MAINTAINER Romin Irani ([email protected]) RUN apt-get update RUN apt-get install -y nginx ENTRYPOINT [“/usr/sbin/nginx”,”-g”,”daemon off;”] EXPOSE 80
![Page 18: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/18.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
Build-Dockerfile
▪dockerbuild-tindigodatacloudapp/ambertools:1.0.0
!
▪AutomatedbuildonDockerHub
18
![Page 19: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/19.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
ShipandRun
▪ShipaDockerImage
• dockerpushindigodatacloudapp/ambertools
!
▪FetchaDockerImage
•dockerpullindigodatacloudapp/ambertools
▪downloaddockerimagefromDockerHubtolocalDockerrepository
!
▪Runadockercontainer• dockerrun[…]indigodatacloudapp/ambertools
- createsadockercontaineroutofthedockerimage
19
![Page 20: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/20.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
OrchestratorAPIs
▪Createadeployment:
• POSTrequestto/deployments-parameters:
- template:stringcontainingaTOSCAYAML-formattedtemplate
- parameters:theinputparametersofthedeployment(mapofstrings)
▪Getdeploymentdetails
• GETrequestto/deployments:
- curl'http://localhost:8080/deployments/<uuid>'
▪Deletedeployment
• DELETErequest
- curl‘http://localhost:8080/deployments/<uuid>'
▪Documentation:http://indigo-dc.github.io/orchestrator/restdocs/#overview
20
![Page 21: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/21.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
▪SubmitaJob:
• POSTrequestto/scheduler/iso8601
- acceptsJSONbody
▪Addadependentjob• POSTto/scheduler/dependency
- acceptJSONhash
▪Listjobs• GETtoscheduler/jobs
- filteronname:GET/scheduler/jobs/search?name=<jobId>
▪Deleteajob• POSTto/scheduler/job/<jobId>
ChronosJobdefinitionandAPIs
21
Exam
ple
![Page 22: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/22.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
INDIGOTOSCAcustomtypesfordescribingChronosjobs
22
The Orchestrator interprets the TOSCA
template and orchestrates the
deployment on Chronos
![Page 23: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/23.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
INDIGOChronosJob:TOSCATemplates
▪ScenarioI:thejobdataarestagedin/outusingtheuser’sonedataspaces(providingproperaccesstoken(s))
• exampletemplate:https://github.com/indigo-dc/tosca-types/blob/master/examples/indigo_job_onedata.yaml
▪ScenarioII:thejobdataarefetchedfrompublicURLsanduploadedtoarepository(web,swift/S3,etc.)usingthecredentialsspecifiedbytheuser
• exampletemplate:https://github.com/indigo-dc/tosca-types/blob/master/examples/indigo_job_output_upload_swift.yaml
23
![Page 24: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/24.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
ChronosJob:ParameterSweep
▪TheTOSCAtemplatedescribesmultiplejobs
▪Eachjobisrunwithaspecificsetofinputparameters
▪ThejobsareruninparallelontheMesoscluster
• thescalingserviceensuresthatnewslavenodesareaddedtothecluster,ifneeded.
24
![Page 25: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/25.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
DemosOutline
▪ ElasticGalaxyCluster(Recorded)Demo:
▪ aGalaxyportalisautomaticallydeployedfromTOSCAandconfiguredtouseaSLURMelasticcluster
▪ workernodesareaddedassoonasjobsaresubmittedthroughtheportal
▪ High-AvailableMesosCluster(Recorded)Demo:
▪ acompleteHAMesosclusterwithChronos/MarathonframeworkisautomaticallydeployedfromaTOSCAtemplate
▪ Applicationexecution(Live)Demos:
▪ Amber
▪ EMSO-INGV(Rscript)
▪ Lifewatch(barridocker)
25
![Page 26: INDIGO PAAS TUTORIAL - Istituto Nazionale di Fisica …agenda.ct.infn.it/event/1216/session/14/material/0/0.pdfe-Research Summer Hackfest - 04 July 2016, Catania INDIGO PAAS Tutorial](https://reader036.vdocument.in/reader036/viewer/2022062909/5afe17dd7f8b9a8b4d8e7f4c/html5/thumbnails/26.jpg)
e-Research Summer Hackfest - 04 July 2016, Catania
DemoOutline(cont.)
▪Long-RunningServicedeploymentDemo:
• deployLong-RunningServices:nginx(webservice),rstudio
• auto-scalingPoC
• load-balancing
26