opening the path to technical excellence

ONE Conference 2013 - 26th September 2013

J.Farres – European Space Agency

R&D for Earth Observation Ground Segment

OpenNebula Conference, Berlin

26/9/2013

Cloud Computing in Space:Opening the path to technical excellence


Agenda

1.Background and Objectives

2.ESA Experiences

1. EO Re-processing on Amazon

2. Dissemination and Processing on Hetzner

3. SuperSites Exploitation platform with Helix Nebula

4. A sandbox service for Science

3.Summary of lessons learnt

4.Future prospects

Objectives (1)


3- Processing bursting

2- Dissemination peaks

1- ICT Costs savings

4- Collaboration platform

Cloud Computing

IaaS

SaaS

Hosting(VPS, Rental)

CDN

PaaS

A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction [NIST]

Background: ESA Mission Statement

ESA's purpose shall be to provide for, and to promote, for exclusively peaceful purposes, cooperation among European States in space research and technology and their space applications, with a view to their being used for scientific purposes and for operational space applications systems:

• by elaborating and implementing a long-term European space policy, by recommending space objectives to the Member States, and by concerting the policies of the Member States with respect to other national and international organisations and institutions;

• by elaborating and implementing activities and programmes in the space field;

• by coordinating the European space programme and national programmes, and by integrating the latter progressively and as completely as possible into the European space programme, in particular as regards the development of applications satellites;

• by elaborating and implementing the industrial policy appropriate to its programme and by recommending a coherent industrial policy to the Member States.


Objectives (1)


3- Processing bursting

2- Dissemination peaks

1- ICT Costs savings

4- Collaboration platform

Cloud Computing

IaaS

SaaS

Hosting(VPS, Rental)

CDN

PaaS

A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction [NIST]

5- Lead effective use of modern computing infrastructures by European industry


Agenda


2.ESA Experiences

1. Mass processing on Amazon





4.Future prospects

Case 1: Mass processing on Amazon(1)

Purpose•Fast re-processing of large EO products collections for CalVal purposes.

Project / Service•Timeframe: 2009 and 2011

•Provider: Amazon, EC2, S3

•Data: ERS SAR Wave, MIPAS (30,000 products)

•System: 200 Virtual Servers

configured as Working Nodes to an ESA grid.

•Usage: 11 CPU years of processing in 5 weeks



https://earth.esa.int/web/guest/missions/esa-operational-eo-missions/envisat/instruments/mipas

Pros Cons


Case 1: Mass processing on Amazon(2)

Pros1. Excellent processing scalability

2. Efficient bulk-in/out data service (via HD)

3. “The faster the cheaper” as it reduces storing costs

4. Good application portability for gridified applications

Cons1. Complex and changing pricing,

e.g. periods with cheaper hosts and free data upload.

2. In-bound / Out-bound costs

3. Ad-hoc scripts to command provisioning

4. AMI format portability


Agenda


2.ESA Experiences






4.Future prospects

Case 2: Dissemination and Processing on Hetzner (1)

Purpose•Couple large processing and dissemination capabilities for low cost.

Project / Service•Timeframe: 2011

•Provider: Hetzner

•Data: 60TB

•System: 1 Head: Catalogue, Processor Register

n Nodes: Data Dissemination, Hadoop Processing Cluster

Packaged as a back-end for web portal services

•Usage: GeoHazards SuperSites (38,000 SAR images and 3,000 users)


InSAR Processing

~150 Satellite images:1.5TB

Time Processing:

150h

CNR

Pros Cons


Case 2: Dissemination and Processing on Hetzner (2)

Pros1. Good archive scalability (chunks

of 8TB)

2. Synergy of processing -dissemination: processing peaks followed by dissemination peaks

3. Much cheaper than Amazon services. No in-boud/out-boud costs

4. Physical dedicated servers enabled easier security

Cons1. System scales storage,

dissemination and processing capabilities simultaneously.

2. Lower service levels than Level 3 or Amazon

3. Virtualization layer had to be deployed (KVM and ONE)


Agenda


2.ESA Experiences






4.Future prospects

Case 3: Exploitation platform with Helix Nebula (1)

Purpose•Pilot a collaborative platform for EO exploitation using multisourced cloud provisioning.

Project / Service•Timeframe: 2012-2013

•Providers: ATOS, CloudSigma, Interoute, T-Systems, EGI

•Cloud Brokers: SlipStream, Entratius

•Data: GeoHazards data (ESA, CNES, DLR, …)

•Processors: ESA, CNR, Gamma, …

•System: 15 TB of raw data, 4 processing services, dedicated VM for selected users, >200 users


Model

EO Data

Software

ICT

Exploitation Platform

Science

Services

Services

ESA UNCLASSIFIED – For Internal Use – Version 2013/04/29

SSEPSuperSites Exploitation Platform

ESA UNCLASSIED - For Offical Use 29/04/2013

04/08/2023

18

ESA PoC on EGI Federated Cloud

The ESA Proof of Concepts on the EGI Federated Cloud focuses on demonstrating the possibility to provide Processing Services to ESA scientists using the EGI Federated Cloud resources

• Participants:– User community

• ESA Research and Service Support section: Configuration and execution of the tests

– Technology providers• Sixsq: Provided Open Source SlipStream software and OCCI connector

– Resource providers• CESNET (OpenNebula): Performance tests and multi-site tests

• GRNET (synnefo): Multi-site tests

• CESGA (OpenNebula): Hosting of the SlipStream server

Pros Cons


Case 3: Exploitation platform with Helix Nebula (2)

Pros1. High performance storage and

dissemination from cloud

2. Scalable processing co-located with data (same VDC)

3. Multi-sourced via cloud brokering services

4. Direct provision to virtual hosts to users in the cloud, via ESA

5. Easy application deployment via grid controller

Cons1. Frequent platform upgrades

2. Need to distribute processing resources near distributed data

3. No Cloud federation. Limitations of brokers

4. COTS licensing

5. Grid on Cloud approach. Workaround to high effort in application “cloudification”


Agenda


2.ESA Experiences






4.Future prospects

Case 4: A sandbox service for Science (1)

Purpose• Provide researchers and service providers a development environment for cloudifying and exploiting their algorithms/services.

Project / Service•Timeframe: 2013-2014

•Providers: Private cloud + Helix Nebula

•Data: Multiple reference data sets from ESA archives

•Processors: Those developed by the users



04/08/2023

22

Many Service

Providers

Pros Cons



Pros1. Hybrid cloud model in support of

development (private) -> deployment (public)

2. Deployment model via PaaS and SaaS

3. Simplified “cloudification” via Cloudera + supporting tools (for SPMD paradigm)

Cons1. Need for new CSP drivers for

Helix Nebula: T-Systems (Zimory), Interoute (Jclouds)

2. Limited CSP support to PaaS and SaaS services.

3. Slow adoption of cloud-reduce paradigm among application developers in Remote Sensing

Role of OpenNebula on the four Cases

Case ONE

1: Mass processing on Amazon

Use of GridWay to command IaaS provisioning from Amazon

2: Dissemination and Processing on Hetzner

Use of ONE to host private IaaS on dedicated hosting services

3: Exploitation platform with Helix Nebula

Use of ONE in resource providers to EGI Federated Cloud

4: A sandbox service for Science

Use of ONE for hybrid cloud provision ESA + Helix NebulaDevelopment of drivers for Zimory and Jclouds



Agenda


2.ESA Experiences






4.Future prospects

Lessons Learnt: ICT provisioning

1. As soon as ICT needs can be predicted and planned, IaaS is more expensive than other hosting solutions like (rental, dedicated hosting).

2. Flexibility of Public IaaS is less appealing when internal resources are pooled, virtualized and managed as an internal cloud.

3. On the other hand, IaaS services allow to size down internal ICT resources to the “fixed” need and ensure their maximum utilization; e.g. using external provisioning for the “variable” need.

Þ Hybrid ICT provisioning


Lessons Learnt: Service Levels

• Terms & Conditions in Public Clouds express surprising low commitment.

• Cloud opportunities can become risks when applied to critical systems.

Þ Develop multi-sourcing

Þ Plan contingency scenarios for services hosted in Public Clouds


Lessons Learnt: Application Areas

• Dissemination and on-demand processing– because are very variable (depending on user demand)

• Secondary archive and re-processing– because are limited in time

• Temporary resources for integration, testing and demonstration– because are limited in time

• System sizing– because needs are unknown

Þ Important areas when remote sensing services can gain from Cloud Computing


Lessons learnt: User expectations

• Open Data– All data are discoverable, accessible online and free

– Data is arranged on long time series of coherent data from different providers.

• Open Computing– Users will be able to perform processing directly on the cloud using virtual servers.

– Users can choose their preferred cloud provider

• Open Source Software– All basic/platform software is open and freely available

– Application can be easily ported across clouds

• Open Collaboration– Data and applications can be easily shared with other users

Þ Be up to the users expectations


Lessons learnt and OpenNebula


OK, Case 4

OK, Case 3

Lesson learnt ONE

Hybrid cloud OK, Case4: A sandbox service for Science

Cloud multi-sourcing OK, Case3: Exploitation platform with Helix Nebula

Application Areas N/A

User Expectations OK, Open Source Software


Agenda


2.ESA Experiences






4.Future prospects

Future Prospects (1)

ICT

• Set-up mid-term relation with 2-4 cloud providers; similarly to current agreements with network providers.

• Cloudify present corporate computing resources

• Establish common ICT provisioning service based on:– Hybrid and multi-sourced resources

– Based on in-house brokering layer



POLICY

• Mandate cloud hosting for specific activities: Integration & Validation, Demonstrators, R&D …

• Promote the use of cloud computing solutions among ESA service providers, in benefit for their competiveness

• Continue to launch specific flagship projects based on public clouds

– SuperSites exploitation platform (continuation)

– Thematic Exploitation platforms



1. Adopt and promote Open data policy– ESA Data Policy

– Data access agreements

2. Adopt and promote interoperability Standards– Data discovery & ordering

– Processing discovery & ordering

– Data / Results access

– RAAA – Registration, Authentification, Authorisation, Accounting

3. Provide Open Source Software– EO Toolboxes

– Data discovery/catalogue tools

– EO Data management and access

4. Promote Open computing infrastructures– Cloud computing paradigm

– Standards for IaaS and brokering services

– Software Licensing Agreements

In summary: Manifest

+ Open Data

+ Open Software

+ Open Computing

=

Increased market development & industrial competiveness
