using a widely distributed federated cloud system to support multiple disparate user communities
TRANSCRIPT
www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323
Using a Widely Distributed Federated Cloud System to Support Multiple
Disparate User Communities
David Wallom
University of Oxford
Chair, EGI Federated Cloud
www.egi.euEGI-InSPIRE RI-261323
Overview
• Why• What• Who
www.egi.euEGI-InSPIRE RI-261323
Growth of Providers
GridLimited academic resource providers
CloudDiversity and scale of new resource providers
www.egi.euEGI-InSPIRE RI-261323
Growth of Userbase
20k researchers 2M researchers!
Few related use cases
Many diverse use cases
www.egi.euEGI-InSPIRE RI-261323
HardwareHardware
HardwareHardware
Hardware
Cloud ManagementCloud Management
Cloud ManagementCloud Management
Cloud Management
User CommunitiesUser Communities
User Communities
Federated interfaces
Federated services
• Open Standards
• Technology independent
• Provider agnosticism
• Production integration and reuse
Principles of Federation
www.egi.euEGI-InSPIRE RI-261323
EGI Cloud Infrastructure
6
EGI Core Platform
Federated AAIServiceRegistry Monitoring Accounting
EGI Cloud Infrastructure Platform
Instance Mgmt
Information Discovery
Storage Management
Cloud Management Stacks(OpenStack, OpenNebula, Synnefo, …)
Help and Support
Security Co-ordination
Training and Outreach
EGI C
olla
bora
tion
Tool
s
EGI A
pplic
ation
D
BIm
age
Repo
sito
ryEG
I Clo
ud S
ervi
ce M
arke
tpla
ce
Sustainable Business Models
User Community
GSIGLUE2
OCCI CDMI
SAM UR
OVF
Cloudinit
www.egi.euEGI-InSPIRE RI-261323
Compatibility for technology providers
Cloud Mgmt. Fram. Fed. AAI Monitoring Accounting Img. Mgmt. OCCI CDMI
OpenStack Yes Yes* Yes Yes Yes Yes
OpenNebula Yes Yes* Yes Yes Yes Yes
Synnefo Yes Yes Yes - Yes - Cloudstack - - - - Yes -
Emotive Yes Yes* - N/A N/A Yes
SlipStream Yes N/A N/A Yes Yes Yes*
Perun Yes Yes N/A N/A N/A N/A
7
Example documentation for a CMF:-https://wiki.egi.eu/wiki/Fedcloud-tf:ResourceProviders:OpenStack
www.egi.euEGI-InSPIRE RI-261323
Geographical dispersion
• 12 countries provide 19 certified resources
– Czech Republic, Germany, Greece, Hungary, Italy, Macedonia, Poland, Slovakia, Spain, Sweden, Turkey, United Kingdom
• 2 countries currently integrating– France, Finland, Portugal, Croatia
• 5 countries interested– Bulgaria, Israel*, The Netherlands,
Switzerland
• Worldwide interest– South Africa* (SAGrid)– South Korea* (KISTI)– United States* (NIST, NSF Centres)
* Not shown on map
www.egi.euEGI-InSPIRE RI-261323
Utilisation
9
217352 VMs launched across 25 clouds
www.egi.euEGI-InSPIRE RI-261323
Federated Cloud Services
Federated IaaS and STaaS Cloud
EYR TC monthly meeting 10
Tier 1:Reliable
Infrastructure Cloud
Tier 4:Zero ICT
Infrastructures
Tier 3:Platform as a Service
Tier 2:General-purpose platform services
PaaS
PaaS
DB
aaS
Had
oop
aaS
VRE
Secure storage
Key
Mgm
t
Encr
yptio
n
ACL
mgm
t
Virtual eLaboratory
17 Feb 2014
www.egi.euEGI-InSPIRE RI-261323
AppDB – Virtual Appliance catalogue
11
www.egi.euEGI-InSPIRE RI-26132312
EGI FedCloud Communities 9/2014
• Ecology – BioVeL: Biodiversity Virtual e-Laboratory
• Structural biology – WeNMR: a worldwide e-Infrastructure for NMR and structural biology
• Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb)
• Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for
volcano and earthquakes monitoring (Collaboration with Helix Nebula)
• Software Engineering – SCI-BUS: simulated environments for portal testing
• Software Engineering – DIRAC: deploying ready-to-use distributed computing systems
• Software Engineering – Catania Science Gateway Framework
• Musicology – Peachnote: dynamic analysis of musical scores
• Earth Observation – ENVRI: Common Operations of Environmental Research
infrastructures (collaboration with EISCAT3D)
• Geology – VERCE: Virtual Earthquake and seismology Research
• Ecology – LifeWatch: E-Science European Infrastructure for Biodiversity and Ecosystem
Research
• High Energy Physics – CERN ATLAS: ATLAS processing cluster via HelixNebula
More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users
www.egi.euEGI-InSPIRE RI-26132313
New EGI FedCloud Communities since launch
• Education – Cranfield University distributed systems course
• Cultural Heritage – DCH-RP management of preservation services in the cloud
• Hydrological Modelling – Running Hydrological models to support real time analysis
• Bioinformatics – ELIXIR execution of the Ensamble application in the Federated Cloud
environment
• Systems implementations – deployment of FTK developed tools and services and data
preservation
• Internet of Things – Smart Grid systems investigation
• Software Development – deployment of research PaaS
• RNA Sequencing – deployment of analysis engines in the cloud
• Physiological Modelling – Calibration, scenario mapping and development
More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users
www.egi.euEGI-InSPIRE RI-26132314
WeNMR e-Infrastructure (VirtualCing)
EGI Federated Cloud Infrastructure
…
NMR Protein DB
ToPoS
Virtual CINGVirtual
CINGVirtual Cing
Virtual CINGVirtual
CINGVirtual Cing
Virtual CINGVirtual
CINGVirtual Cing
OCCI client
Virtual Common Interface for NMR structure generation (VirtualCing), developed in WeNMR, has been deployed via the EGI Federated Cloud
www.egi.euEGI-InSPIRE RI-261323
ESA HN Flagship on the EGI Federated Cloud
The European Space Agency deployed part of the ESA Helix Nebula Flagship into the EGI Federated Cloud:
1. Use SlipStream Infrastructure Broker to deploy a custom computing cluster on multiple Federated Cloud sites (CESNET, GRNET)
2. Connect the computing cluster to the ESA G-POD system and run SAR processing for volcano and earthquake monitoring
15
Automatic Cluster Deployment
EGI Federated CloudComputing Cluster
Cluster Head
Working Nodes
OCCI
Globus
Job submission
BLUE BOX
www.egi.euEGI-InSPIRE RI-26132316
EISCAT 3D PoC
EISCAT_3D is a three-dimensional imaging radar to be located in the northernmost parts of Europe, to study Earth atmosphere and its coupling to space.
The EISCAT 3D Proof of Concept is a PaaS built on top of the EGI Federated Cloud for Data Access and Dissemination, based on the ENVRI tools, the EGI Federated Cloud Infrastructure and Storage services and the current EISCAT data.
www.egi.euEGI-InSPIRE RI-261323
Peachnote
04/15/2023 17
Peachnote is a music score search engine and analysis platform.
Hundreds of thousands of music scores are being digitized by libraries all over the world. In contrast to books, they generally remain inaccessible for content-based retrieval and algorithmic analysis.
There is no analogue to Google Books for music scores, and no large corpora exists that can empower advanced analysis on music scores.
Peachnote want to help change that providing visitors and researchers access to a massive amount of symbolic music data.
EGI Federated Cloud
OMR Worker
PDF Splitter
OMR Feeder
OMR Worker
…
www.egi.euEGI-InSPIRE RI-261323
OpenModeller on the Biovel Portal
04/15/2023 18
The Ecological Niche Modeling (ENM) Workflow takes as input a file containing species occurrence points to create a model with the openModeller Web Service.
• The EUBrazilOpenBio ENM service is exposed through an extended openModeller Web Service interface• Multi-staging and multi-parametric oM experiments are implemented through COMPSs that dynamically creates the virtual resources to execute the operations.• An OCCI connector is used for the VMs management while data management supports CDMI endpoints.
ENM Service (OMWS2)
VENUS-C Cloud Middleware
COMPSs Workflow Orchestrator
COMPSs Workflow OrchestratorOCCIOCCI CDMICDMI
EGI Federated Cloud
Service available athttps://portal.biovel.eu/
www.egi.euEGI-InSPIRE RI-26132304/15/2023 20
Running GNU Octave and R on the CHAIN-REDS Science Gateway
20
CHAIN-REDS Science Gateway-CHAIN-REDS RI-306819
www.egi.euEGI-InSPIRE RI-261323
Strengthening the underpinning platform
Technology Evolution• Broader support for open standards in Cloud
management frameworks– Utilisation of rOCCI for interfaces to commercial cloud
frameworks
• New feature additions to foundational tools depending on requests– Accounting, monitoring, service discovery, Application DB
• Innovative deployments of opensource PaaS• Large improvements on documentation for all 3
stakeholder groups
22
Valu
e ad
ded
serv
ices
for U
ser
Com
mun
ities
www.egi.euEGI-InSPIRE RI-261323
ConclusionsThe EGI Federated Cloud, a federation of institutional private Clouds, offering Cloud
Services to researchers in Europe and worldwide• A single cloud system able to
– Scale to user needs– Integrate multiple different providers to give resilience– Prevent vendor lock-in through open standards– Enable resource provision targeted towards the research community
• Paving the way for a global federated cloud marketplace– Established best practice– Illustrating leadership– Open standards, open technology– Open membership, open processes– Open competition
Current capability – ~6,000 cores, 300 TB storage
Promised through EGI-Inspire – 18,000 cores, 6000 TB storage
2020 Target – 1M cores, 1EB storage
23
Alison Packer, Álvaro López García, Alvaro Simon, Binh Minh Nguyen, Björn Hagemeier, Boris Parak, Boro Jakimovski, Cal
Loomis, Christos Loverdos, Daniele Cesini, Daniele Lezzi, David Blundell, Diego Scardaci, Elisabetta Ronchieri, Emir Imamagic, Enol Fernandez, Feyza Eryol, Florian Feldhaus, Gergely Sipos,
Ivan Diaz, Jan Meizner, John Gordon, Kostas Koumantaros, Malgorzata Krakowian, Marios Chatziangelou, Marco Verlato, Marica Antonacci , Mattieu Puel, Matteo Turilli, Michel Jouvin, Michel Drescher, Miroslav Ruda, Nuno L. Ferreira, Owen Synge,
Paul Miller, Peter Solagna, Piotr Kasprzak , Roberto Barbera, Ruben Valles, Sándor Ács, Salvatore Pinto, Silvio Spardi,
Soonwook Hwang, Steven Newhouse, Stuart Pullinger, Thijs Metsch, Tomasz Szepieniec, Viet Tran, Zeeshan Ali Shah and all
the other members of the EGI federated Cloud Task Force, activity, workpackage
www.egi.euEGI-InSPIRE RI-261323
Thank you!
Questions?
25
Task Force resources
Mailing List: [email protected]
Wiki site: http://go.egi.eu/tf-fedclouds
GitHub: https://github.com/EGI-FCTF
Indico site: https://www.egi.eu/indico/categoryDisplay.py?categId=112