opennebulaconf 2013 - keynote: opening the path to technical excellence by jordi farres
Post on 22-Nov-2014
1.704 Views
Preview:
DESCRIPTION
TRANSCRIPT
Page 1ONE Conference 2013 - 26th September 2013
J.Farres – European Space Agency
R&D for Earth Observation Ground Segment
OpenNebula Conference, Berlin
26/9/2013
Cloud Computing in Space:Opening the path to technical excellence
Page 2ONE Conference 2013 - 26th September 2013
Agenda
1.Background and Objectives
2.ESA Experiences
1. EO Re-processing on Amazon
2. Dissemination and Processing on Hetzner
3. SuperSites Exploitation platform with Helix Nebula
4. A sandbox service for Science
3.Summary of lessons learnt
4.Future prospects
Page 3
Objectives (1)
ONE Conference 2013 - 26th September 2013
3- Processing bursting
2- Dissemination peaks
1- ICT Costs savings
4- Collaboration platform
Cloud Computing
IaaS
SaaS
Hosting(VPS, Rental)
CDN
PaaS
A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction [NIST]
Page 4
Background: ESA Mission Statement
ESA's purpose shall be to provide for, and to promote, for exclusively peaceful purposes, cooperation among European States in space research and technology and their space applications, with a view to their being used for scientific purposes and for operational space applications systems:
• by elaborating and implementing a long-term European space policy, by recommending space objectives to the Member States, and by concerting the policies of the Member States with respect to other national and international organisations and institutions;
• by elaborating and implementing activities and programmes in the space field;
• by coordinating the European space programme and national programmes, and by integrating the latter progressively and as completely as possible into the European space programme, in particular as regards the development of applications satellites;
• by elaborating and implementing the industrial policy appropriate to its programme and by recommending a coherent industrial policy to the Member States.
ONE Conference 2013 - 26th September 2013
Page 5
Objectives (1)
ONE Conference 2013 - 26th September 2013
3- Processing bursting
2- Dissemination peaks
1- ICT Costs savings
4- Collaboration platform
Cloud Computing
IaaS
SaaS
Hosting(VPS, Rental)
CDN
PaaS
A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction [NIST]
5- Lead effective use of modern computing infrastructures by European industry
Page 6ONE Conference 2013 - 26th September 2013
Agenda
1.Background and Objectives
2.ESA Experiences
1. Mass processing on Amazon
2. Dissemination and Processing on Hetzner
3. SuperSites Exploitation platform with Helix Nebula
4. A sandbox service for Science
3.Summary of lessons learnt
4.Future prospects
Page 7
Case 1: Mass processing on Amazon(1)
Purpose•Fast re-processing of large EO products collections for CalVal purposes.
Project / Service•Timeframe: 2009 and 2011
•Provider: Amazon, EC2, S3
•Data: ERS SAR Wave, MIPAS (30,000 products)
•System: 200 Virtual Servers
configured as Working Nodes to an ESA grid.
•Usage: 11 CPU years of processing in 5 weeks
ONE Conference 2013 - 26th September 2013
Page 8ONE Conference 2013 - 26th September 2013
https://earth.esa.int/web/guest/missions/esa-operational-eo-missions/envisat/instruments/mipas
Page 9
Pros Cons
ONE Conference 2013 - 26th September 2013
Case 1: Mass processing on Amazon(2)
Pros1. Excellent processing scalability
2. Efficient bulk-in/out data service (via HD)
3. “The faster the cheaper” as it reduces storing costs
4. Good application portability for gridified applications
Cons1. Complex and changing pricing,
e.g. periods with cheaper hosts and free data upload.
2. In-bound / Out-bound costs
3. Ad-hoc scripts to command provisioning
4. AMI format portability
Page 10ONE Conference 2013 - 26th September 2013
Agenda
1.Background and Objectives
2.ESA Experiences
1. Mass processing on Amazon
2. Dissemination and Processing on Hetzner
3. SuperSites Exploitation platform with Helix Nebula
4. A sandbox service for Science
3.Summary of lessons learnt
4.Future prospects
Page 11
Case 2: Dissemination and Processing on Hetzner (1)
Purpose•Couple large processing and dissemination capabilities for low cost.
Project / Service•Timeframe: 2011
•Provider: Hetzner
•Data: 60TB
•System: 1 Head: Catalogue, Processor Register
n Nodes: Data Dissemination, Hadoop Processing Cluster
Packaged as a back-end for web portal services
•Usage: GeoHazards SuperSites (38,000 SAR images and 3,000 users)
ONE Conference 2013 - 26th September 2013
Page 12
InSAR Processing
~150 Satellite images:1.5TB
Time Processing:
150h
CNR
Page 13
Pros Cons
ONE Conference 2013 - 26th September 2013
Case 2: Dissemination and Processing on Hetzner (2)
Pros1. Good archive scalability (chunks
of 8TB)
2. Synergy of processing -dissemination: processing peaks followed by dissemination peaks
3. Much cheaper than Amazon services. No in-boud/out-boud costs
4. Physical dedicated servers enabled easier security
Cons1. System scales storage,
dissemination and processing capabilities simultaneously.
2. Lower service levels than Level 3 or Amazon
3. Virtualization layer had to be deployed (KVM and ONE)
Page 14ONE Conference 2013 - 26th September 2013
Agenda
1.Background and Objectives
2.ESA Experiences
1. Mass processing on Amazon
2. Dissemination and Processing on Hetzner
3. SuperSites Exploitation platform with Helix Nebula
4. A sandbox service for Science
3.Summary of lessons learnt
4.Future prospects
Page 15
Case 3: Exploitation platform with Helix Nebula (1)
Purpose•Pilot a collaborative platform for EO exploitation using multisourced cloud provisioning.
Project / Service•Timeframe: 2012-2013
•Providers: ATOS, CloudSigma, Interoute, T-Systems, EGI
•Cloud Brokers: SlipStream, Entratius
•Data: GeoHazards data (ESA, CNES, DLR, …)
•Processors: ESA, CNR, Gamma, …
•System: 15 TB of raw data, 4 processing services, dedicated VM for selected users, >200 users
ONE Conference 2013 - 26th September 2013
Page 16
Model
EO Data
Software
ICT
Exploitation Platform
Science
Services
Services
ESA UNCLASSIFIED – For Internal Use – Version 2013/04/29
Page 17
SSEPSuperSites Exploitation Platform
ESA UNCLASSIED - For Offical Use 29/04/2013
Page 18
04/08/2023
18
ESA PoC on EGI Federated Cloud
The ESA Proof of Concepts on the EGI Federated Cloud focuses on demonstrating the possibility to provide Processing Services to ESA scientists using the EGI Federated Cloud resources
• Participants:– User community
• ESA Research and Service Support section: Configuration and execution of the tests
– Technology providers• Sixsq: Provided Open Source SlipStream software and OCCI connector
– Resource providers• CESNET (OpenNebula): Performance tests and multi-site tests
• GRNET (synnefo): Multi-site tests
• CESGA (OpenNebula): Hosting of the SlipStream server
Page 19
Pros Cons
ONE Conference 2013 - 26th September 2013
Case 3: Exploitation platform with Helix Nebula (2)
Pros1. High performance storage and
dissemination from cloud
2. Scalable processing co-located with data (same VDC)
3. Multi-sourced via cloud brokering services
4. Direct provision to virtual hosts to users in the cloud, via ESA
5. Easy application deployment via grid controller
Cons1. Frequent platform upgrades
2. Need to distribute processing resources near distributed data
3. No Cloud federation. Limitations of brokers
4. COTS licensing
5. Grid on Cloud approach. Workaround to high effort in application “cloudification”
Page 20ONE Conference 2013 - 26th September 2013
Agenda
1.Background and Objectives
2.ESA Experiences
1. Mass processing on Amazon
2. Dissemination and Processing on Hetzner
3. SuperSites Exploitation platform with Helix Nebula
4. A sandbox service for Science
3.Summary of lessons learnt
4.Future prospects
Page 21
Case 4: A sandbox service for Science (1)
Purpose• Provide researchers and service providers a development environment for cloudifying and exploiting their algorithms/services.
Project / Service•Timeframe: 2013-2014
•Providers: Private cloud + Helix Nebula
•Data: Multiple reference data sets from ESA archives
•Processors: Those developed by the users
ONE Conference 2013 - 26th September 2013
Page 22
Case 4: A sandbox service for Science (2)
04/08/2023
22
Many Service
Providers
Page 23
Pros Cons
ONE Conference 2013 - 26th September 2013
Case 4: A sandbox service for Science (3)
Pros1. Hybrid cloud model in support of
development (private) -> deployment (public)
2. Deployment model via PaaS and SaaS
3. Simplified “cloudification” via Cloudera + supporting tools (for SPMD paradigm)
Cons1. Need for new CSP drivers for
Helix Nebula: T-Systems (Zimory), Interoute (Jclouds)
2. Limited CSP support to PaaS and SaaS services.
3. Slow adoption of cloud-reduce paradigm among application developers in Remote Sensing
Page 24
Role of OpenNebula on the four Cases
Case ONE
1: Mass processing on Amazon
Use of GridWay to command IaaS provisioning from Amazon
2: Dissemination and Processing on Hetzner
Use of ONE to host private IaaS on dedicated hosting services
3: Exploitation platform with Helix Nebula
Use of ONE in resource providers to EGI Federated Cloud
4: A sandbox service for Science
Use of ONE for hybrid cloud provision ESA + Helix NebulaDevelopment of drivers for Zimory and Jclouds
ONE Conference 2013 - 26th September 2013
Page 25ONE Conference 2013 - 26th September 2013
Agenda
1.Background and Objectives
2.ESA Experiences
1. EO Re-processing on Amazon
2. Dissemination and Processing on Hetzner
3. SuperSites Exploitation platform with Helix Nebula
4. A sandbox service for Science
3.Summary of lessons learnt
4.Future prospects
Page 26
Lessons Learnt: ICT provisioning
1. As soon as ICT needs can be predicted and planned, IaaS is more expensive than other hosting solutions like (rental, dedicated hosting).
2. Flexibility of Public IaaS is less appealing when internal resources are pooled, virtualized and managed as an internal cloud.
3. On the other hand, IaaS services allow to size down internal ICT resources to the “fixed” need and ensure their maximum utilization; e.g. using external provisioning for the “variable” need.
Þ Hybrid ICT provisioning
ONE Conference 2013 - 26th September 2013
Page 27
Lessons Learnt: Service Levels
• Terms & Conditions in Public Clouds express surprising low commitment.
• Cloud opportunities can become risks when applied to critical systems.
Þ Develop multi-sourcing
Þ Plan contingency scenarios for services hosted in Public Clouds
ONE Conference 2013 - 26th September 2013
Page 28
Lessons Learnt: Application Areas
• Dissemination and on-demand processing– because are very variable (depending on user demand)
• Secondary archive and re-processing– because are limited in time
• Temporary resources for integration, testing and demonstration– because are limited in time
• System sizing– because needs are unknown
Þ Important areas when remote sensing services can gain from Cloud Computing
ONE Conference 2013 - 26th September 2013
Page 29
Lessons learnt: User expectations
• Open Data– All data are discoverable, accessible online and free
– Data is arranged on long time series of coherent data from different providers.
• Open Computing– Users will be able to perform processing directly on the cloud using virtual servers.
– Users can choose their preferred cloud provider
• Open Source Software– All basic/platform software is open and freely available
– Application can be easily ported across clouds
• Open Collaboration– Data and applications can be easily shared with other users
Þ Be up to the users expectations
ONE Conference 2013 - 26th September 2013
Page 30
Lessons learnt and OpenNebula
ONE Conference 2013 - 26th September 2013
OK, Case 4
OK, Case 3
Lesson learnt ONE
Hybrid cloud OK, Case4: A sandbox service for Science
Cloud multi-sourcing OK, Case3: Exploitation platform with Helix Nebula
Application Areas N/A
User Expectations OK, Open Source Software
Page 31ONE Conference 2013 - 26th September 2013
Agenda
1.Background and Objectives
2.ESA Experiences
1. EO Re-processing on Amazon
2. Dissemination and Processing on Hetzner
3. SuperSites Exploitation platform with Helix Nebula
4. A sandbox service for Science
3.Summary of lessons learnt
4.Future prospects
Page 32
Future Prospects (1)
ICT
• Set-up mid-term relation with 2-4 cloud providers; similarly to current agreements with network providers.
• Cloudify present corporate computing resources
• Establish common ICT provisioning service based on:– Hybrid and multi-sourced resources
– Based on in-house brokering layer
ONE Conference 2013 - 26th September 2013
Page 33
Future Prospects (2)
POLICY
• Mandate cloud hosting for specific activities: Integration & Validation, Demonstrators, R&D …
• Promote the use of cloud computing solutions among ESA service providers, in benefit for their competiveness
• Continue to launch specific flagship projects based on public clouds
– SuperSites exploitation platform (continuation)
– Thematic Exploitation platforms
ONE Conference 2013 - 26th September 2013
Page 34
Future Prospects (3)
1. Adopt and promote Open data policy– ESA Data Policy
– Data access agreements
2. Adopt and promote interoperability Standards– Data discovery & ordering
– Processing discovery & ordering
– Data / Results access
– RAAA – Registration, Authentification, Authorisation, Accounting
3. Provide Open Source Software– EO Toolboxes
– Data discovery/catalogue tools
– EO Data management and access
4. Promote Open computing infrastructures– Cloud computing paradigm
– Standards for IaaS and brokering services
– Software Licensing Agreements
Page 35
In summary: Manifest
+ Open Data
+ Open Software
+ Open Computing
=
Increased market development & industrial competiveness
ONE Conference 2013 - 26th September 2013
top related