cloud processing close to the earth data

Cloud Processing close to the Earth data

Herve Caumont Terradue Srl.

NextGEOSS KOM - Lisbon, January 18th 2017

Agenda

Foreword

1. A new workplace 2. When Earth data matters 3. Hubs and federations 4. For knowledge producers Final remarks

Foreword Applied research serves society, its funding is focused to solve current or future societal challenges To measure and understand the earth conditions, new and improved numerical models are leading the way, along with improved data assimilation pipelines. How will the Earth Observation community better support this goal ? “major technical challenge is the complexity of the data and analytics procedures across disciplines (rather than the size of the data per se)”

Realising the European Open Science Cloud EC Directorate-General for Research and Innovation

A New Workplace Our road to bring EO data mainstream

1

NextGEOSS is open

NextGEOSS builds heavily on openness, especially Open Data policies, to deliver the European GEO data hub.

Taking heritage of the GEO 2005-2015 achievements, promoting the use of Earth Observations in decision making, and innovating for GEOSS, a set of coordinated, independent EO, information and processing systems.

Looking at the next decade, NextGEOSS will engage with communities by promoting innovative GEOSS powered applications from Europe.

Cloud processing close to the data shall be a fast, cost effective and user friendly capability so that application developers can focus on creating added value products.

Terradue Cloud Platform supports NextGEOSS approach to Cloud

Terradue Cloud Platform

Integration layers: ▪  Connecting scientists to the Web and Cloud resources

empowering their research

▪  Tailored for Earth Sciences data integration & analysis

Partnership mindset: ▪  Dedicated entry points for different types of value adders

Since 2013, we’ve supported more than 40 companies and research institutes in accessing earth observation data & deploying processing services


2009-2014

2015-Forward

ESA G-POD

G-POD Cloud Experiment

GEO Supersites Virtual Archive

EO Thematic Platforms

GENESI-DR/DEC

GEOWOW

MELODIES

H2020 Projects

2006-2008

▪  Co-funded ESA and EC projects, in a common vision ▪  Validation of evolutions via user engagement on the platform,

with network effects carrying business opportunities


2009-2014

2015-Forward

ESA G-POD




GENESI-DR/DEC

GEOWOW

MELODIES

H2020 Projects

2006-2008 Technological innovations, domain standards takeup




2009-2014

ESA G-POD




GENESI-DR/DEC

GEOWOW

MELODIES

H2020 Projects

2006-2008 Technological innovations, domain standards takeup

2015-Forward

Learning from experiments, capacity building






2009-2014

2015-Forward

ESA G-POD




GENESI-DR/DEC

GEOWOW

MELODIES

H2020 Projects

2006-2008

Operations level agreements, user engagement

Technological innovations, domain standards takeup

Learning from experiments, capacity building

Enabling ‘network effects’

What does a Platform mean nowadays ?

✓  A lever of Web and Cloud technologies

✓  A business model for value co-creation

✓  A framework to bring innovation to new or larger communities


A workplace & APIs to connect Data Providers, Developers, Integrators and information producers

Providers

Consumers

▶  Mission:

▶  Rapid prototyping and benchmarking of algorithms

▶  Seamless data access whatever stage of the process

▶  Automated data processing

▶  Scalable resource management

▶  Information conveyance

When Earth data matters The ‘decades of observations’ challenge

2

Long time series for earth science

Earth observation data is a unique source of global measurements over decades

Source: H. Laur, 2nd Workshop European Cloud Computing Strategic Plan Heidelberg, 26 Oct 2011

Source: C.Minet et al., 2012

When calibrated and combined with other sources, it empowers validation & interpolation models

Earth observation categories

Atmosphere

Cryosphere

Source: GMES

Example of societal benefits

Support the enactment of policies

such as the monitoring of greenhouse gas emissions


Document both climate change

trends and new economic opportunities in the Arctic based on

sea ice monitoring

Connect existing activities to NextGEOSS


Geosphere

Hydrosphere

Source: GMES


Estimate erosion rates sending

sediments and nutrients into waterways

Mitigate geohazards risks

impacting human settlements

Examples of societal benefits

Deliver advice to farmers based on weather conditions & soil moisture

Sustain ship routing efficiency

based on ocean conditions



Biosphere

Source: GMES

Examples of societal benefits

Document Essential Biodiversity Variables to improve ecosystem

benefits in face of increasing anthropogenic pressures

Monitor the state and trends of ecosystems in biogeographical

regions (e.g. mountain, arid, semi-arid, coastal and marine) for

internationally recognized protected aras and to define the needs of

future protected areas


Illustration from the GEO Geohazard Supersites

NextGEOSS challenge:

Private sector

❏  overcome the fragmentation of the European landscape of existing public & private Earth Observation data infrastructures

❏  maximise their combined exploitation

Socio-economic open datasets

govdata.de

data.gov.uk

data.gouv.fr

catalog.data.gov

PublicData.eu

Private sector

▪  Address requirements for the development of the European digital economy

▪  Bring together multi-source assets having an open data policies

▪  Help more researchers & developers to compose data-driven applications

NextGEOSS foreseen impact:

Hubs and federations Empowering the commons

3

A federated access for Earth Observations data

Data & processors swarming

Data Staging from distributed repositories

Data packages capturing input needs Deliver new

products and analysis results towards community hubs

Appliance deployment to distributed hosts in the cloud

NextGEOSS Service Suite

Architectures & Innovation

▪  Web and Cloud services ▪  Standard App containers ▪  Mobile technologies ▪  Global Network operations ▪  Open Data Policies

▪  System of systems governance

"the architecture of a complex technical system fundamentally influences, and is influenced by, the economic system that drives the development, production, and use of the system"

Internet Architecture and Innovation MIT Press, Barbara van Schewick

For knowledge producers Addressing value creation chains

4

Concept of Operation

Data access

Research and operational data infrastructures

Copernicus service infrastructures

Within the NextGEOSS consortium:

GEOSS registered resources

Copernicus scientific & public access

NextGEOSS Data Hub

Faceted search

User feedback Data mining

QoS

Data staging

OGC OpenSearch API

Data Buckets API

Enablers for Community hubs

NextGEOSS Data management

Data I/O workflow automation in a

distributed environment

Cloud bursting

NextGEOSS support to community hubs in

requesting, managing and accessing resources for

cloud bursting

Covering both the open science and commercial applications opportunities

Experience with community hubs

5

Climate Change

Carbon market

Climatology

Oceanography

Ship logistics

Land management

Water quality

Desertification

Biodiversity

Urban accounting

Volcanology

Seismic analysis

Disaster Risk Reduction

EC FP7 GEOWOW (2011-2013)

EC FP7 SenSyF (2013-2015)

EC FP7 MELODIES (2014-2016)

ESA Exploitation Platforms (2014 onwards)

8

14

35+

H2020 EcoPotential (2015-2018)

30+

Campaigns (2013 onwards)

NextGEOSS intended outreach through innovative

research Pilots & business opportunity Pilots

Atmosphere

Marine Land

Cold regions Biodiversity

Air pollution & Health Water management

Agriculture / Agronomy Food security

Teritorial planning Solar energy

Urban growth Smart cities

Secure societies Citizen observatories

Final Remarks

With NextGEOSS, state of the art

technology will match a proven

governance model, driven by the

public sector with possible

contributions from private entities,

and deliver to society a new

capacity to learn faster from

earth observations, and make

better informed decisions

References

WMO Global Atmoshere Watch Implementation Plan 2016-2023 http://www.wmo.int/pages/prog/arep/gaw/documents/IPFinalDraftMay11.pdf GSNL 2017-2019 GSNL Implementation plan http://www.earthobservations.org/documents/gsnl/201607_GSNL_Implementation_plan_2017_2019.pdf Realising the European Open Science Cloud, EC Directorate-General for Research and Innovation https://ec.europa.eu/research/openscience/pdf/realising_the_european_open_science_cloud_2016.pdf Internet Architecture and Innovation, MIT Press, Barbara van Schewick https://mitpress.mit.edu/books/internet-architecture-and-innovation

cloud processing close to the earth data

Data & Analytics