Coastal Waters Research Synergy Framework

Coastal Waters Research Synergy Framework

This project has received funding from the European

Union’s Horizon 2020 Research and Innovation Programme under

grant agreement no 687289

Summer School Meeting, Thursday 13th July 2017LNEC, Lisbon, Portugal

Co-ReSyF Hands-on sessions▶ Hervé Caumont (Terradue)

Coastal Waters ResearchSynergy Framework

http://co-resyf.eu/

2

http://geoportal.co-resyf.eu/

Cloud services

User profiles

Novice:- EO data access helpers- Processing ‘as-a-Service’

Intermediate:- Workflow composer service- Reuse of existing modules

Expert:- Cloud Sandbox service- Software integration

❏ Data Processing

❏ Algorithm integration

❏ Collaborative research

3

Research Apps

Aiming at ‘Platform’ services

Front-end Back-end

4

Data Search

Workflow Composer

Workflow Manager

Visualisation & Analysis

Platform Services v1.0

Hosted processor

Third-party tools repos

Co-ReSyF toolkit

repository

Workflow Engine

Data Agency

Cloud Controller

Processor as a Service

User Support

AAA

OGC WPS, Hadoop

OGC WPS

▶ OGC OSxGT

▶ yum, conda

5Innovation & automationBenefits for Platform users

Develop Jobs

Setup Workflow

Sandbox Environment

Validate App

Operational Cloud

Environment

Design Service

Deploy Service

Monitor Service

▶Hands-on focus

6

An application lifecycle

7

▪ Processing libs developed in any programming language supported by CentOS 6.5

▪ C/C++, Java, Python (most common)▪ Matlab and IDL▪ Other languages shall be

previously analyzed

▪ Then synchronized onthe Platform via softwarepackages management tools(RPM repos, GitHub, …)

Design & Libraries Declaration (1/4)

8

Python, CDAT, R, BEAM, BEAT

Service integration (2/4)

▪ Service developer uses the sandbox tools to implement, test and validate a scalable workflow

▪ Sandbox includes tools to proof test data discovery, parallel processing and access to results

9Cloud Compute deployment (3/4)

▪ After validation, the developer publishes the Service for integration on a selected Cloud Infrastructure

▪ A ‘Processor as a Service’ accesses complete pools of datasets,allows application intensive computing, and stores processor outputs, e.g. back to the lab

10Service Operations (4/4)

▪ Processing Services are operated on a selected Cloud provider

▪ Operators manage the service via their user account on Terradue Cloud Platform

▪ Resources can scale up/down▪ The development environment

remains active for maintenance activities on the Service

▪ Service End Users can access the service’s products from ad-hoc community portals

▪ Includes support for standard protocols like OGC Web Services

11

Hands-on sessions

12Hands-on sessions overview

Session 1 - Data access and adding your own data to the platform Cloud storage

What are the Platform APIs, along with integrated tools for data staging within Cloud Apps, that deliver custom data management services to registered developer users on the Platform ?

Session 2 - Develop your own processing chain, using existing modules

How registered users are able to create a work environment with state of the art tools like Git and GitHub, Maven, Oozie workflows, … and work collaboratively for the development of powerful Apps ?

13

Get the instructions

Visit

https://gist.github.com/HerveCaumont/

Select

simple-da-training-part-1

Session 1Data access and adding your own data to the platform Cloud storage

14

Get the instructions

Visit

https://gist.github.com/HerveCaumont/

Select

simple-dcs-training-part-1

Session 2Integrate and test your own processing chain, using existing modules

15

Co-ReSyF Project EC fundinghttp://www.copernicus.eu/projects/co-resyf

Co-ReSyF Websitehttp://co-resyf.eu/

Co-ReSyF GeoPortalhttp://geoportal.co-resyf.eu/

Co-ReSyF Twitter handlehttps://twitter.com/Co_ReSyF

Co-ReSyF repositories on GitHubhttps://github.com/ec-coresyf

User forumhttps://discuss.terradue.com https://discuss.terradue.com/c/co-resyf

Session 3Resources at your disposal

Terradue Cloud Platform, sign-uphttps://www.terradue.com

Private Cloud VPN setuphttps://access.terradue.com

Cloud Dashboardhttps://cloud.terradue.com

Dev Cloud Sandbox (DCS) documentationhttps://docs.terradue.com/developer-sandbox

DCS Community shared applicationshttps://github.com/Terradue?q=dcs

T2 API documentationhttp://docs.terradue.com/t2-api/

Operations Supporthttps://support.terradue.com

Looking forward hearing from you!CO-RESYF SUMMER SCHOOL 2017

16

Hervé CaumontCloud Platform Operations, Terradueherve.caumont@terradue.com

Back-up slides(from RPM-1)

17

Choose a language 18

Sen2Cor is a processor for Sentinel-2 Level 2A product generation and formatting; it performs the atmospheric, terrain and cirrus correction of Top-Of- Atmosphere Level 1C input data.

Sen2Cor creates Bottom-Of-Atmosphere, optionally terrain and cirrus corrected reflectance images; additional, Aerosol Optical Thickness, Water Vapor, Scene Classification Maps and Quality Indicators for cloud and snow probabilities.

http://step.esa.int/main/third-party-plugins-2/sen2cor/

https://anaconda.org/Terradue/sen2cor

19The sen2cor processing tool

20Understand the application structure

function main()

{

local reference=${1}

ciop-log "INFO" "**** Sentinel-2 Atmospheric Correction ****"

ciop-log "INFO" "------------------------------------------------------------"

ciop-log "INFO" "Input S-2 L1C product reference: ${reference}"

ciop-log "INFO" "------------------------------------------------------------"

ciop-log "INFO" "STEP 1: Getting input product"

local_product=$( get_data "${reference}" "${TMPDIR}" ) || return ${ERR_GET_DATA}

ciop-log "INFO" "------------------------------------------------------------"

ciop-log "INFO" "STEP 2: SEN2COR tool" output=$( sen2cor "${reference}" "${local_product}" ) || return ${ERR_GET_DATA}

ciop-log "INFO" "------------------------------------------------------------"

ciop-log "INFO" "STEP 3: Publishing results" publish_data "${output}" || return ${ERR_PUBLISH}

ciop-log "INFO" "------------------------------------------------------------"

}

21Understand the application structure

for reference in sys.stdin:

reference = reference.strip('\n')

ciop.log('INFO', '**** Sentinel-2 Atmospheric Correction ****')

ciop.log('INFO', '------------------------------------------------------------')

ciop.log('INFO', 'Input S-2 L1C product reference: ' + reference)

ciop.log('INFO', '------------------------------------------------------------')

ciop.log('INFO', 'STEP 1: Getting input product')

local_product = get_data(reference, os.environ['TMPDIR'])

ciop.log('INFO', '------------------------------------------------------------')

ciop.log('INFO', 'STEP 2: SEN2COR tool')

output = sen2cor(reference, local_product)

ciop.log('INFO', '------------------------------------------------------------')

ciop.log('INFO', 'STEP 3: Publishing results')

publish_data(output)

ciop.log('INFO', '------------------------------------------------------------')

22Inspect the logs - Job

23Inspect the logs - Task

24Inspect the logs - Attempt