wp10 –virtual research environment – progress of vre ......• virtual code sprint in june 2018...

[email protected] – www.seadatanet.org

WP10 – Virtual Research Environment –

Progress of VRE development

WP10 team


Outline

1. SDC VRE – progress summary

2. Demo of individual components

– Marine-ID and B2Access (Themis, 5 mins)

– WebODV (Sebastian, 10 mins)

– DIVAnd and Notebooks (Charles, 10 mins)

– Biological QC (Simon, 5 mins)

– Visualisation (Giorgio, 5 mins)

3. VRE deadlines and deliverables

2


1. VRE – Progress summary

3


SeaDataClouds VRE needs to:– Facilitate collaborative and individual research:

– Combine data with subsets from other data resources, such as the ingested collections

– Have a high capacity and performance for big data processing and state-of-the-art web visualisation services

4


– Respect privacy of users and differences in data policies. Differentiated users, different access to data and data products.

– Be possible to configure virtual work spaces for individuals or groups to work on specific projects, including setting up of dedicated pools of data

– Allow producers to decide whether their outcomes will be shared in the public domain or stay private

– Be based and hosted on EUDAT’s infrastructure based on it B2-… service platforms

5


Focus on 5 versatile use cases:

1. SeaDataNet Temperature and Salinity water column analysis

2. EMODNET - Chemistry, same for bio-geo-chemistry3. SeaDataNet Biology Quality Assessment4. EMODNET - HRSM, DTM processing5. Processing and visualising data sets

6


SDC VRE Architecture – level 1

7


SDC VRE Architecture – level 2

8


Prototype functions targeted of T/S

(narrow abstract of total use case!)

log in with single sign on B2ACCESS + Marine-ID

integration GUI development Javascript library

apply water column obs quality control with friendly data editor and

save result,

webODV

advise data centre of the regional quality control

be advised of quality control result (email of log of changes/anomalies

sorted per DC)email

configure DIVA interpolation

Jupyter + DIVA library

apply DIVA interpolation, send notification (email) when processing is

completed

visualize interpolation result together with original observations of other

observations

extract and view profiles, time series, hovmuller out of the interpolation

result

publish dataset results (metadata and data), get a DOIoceanBrowser+sextant-

dataCite

9


2. Progress per componentSome highlights

10


Connecting Marine-ID to B2Access - Themis

11


Connecting B2ACCESS - Marine-ID

• B2ACCESS is an easy-to-use and secure

Authentication and Authorization platform

developed by EUDAT. B2ACCESS is versatile and

can be integrated with any service.

• Marine-ID is an integrated Registration,

Authentication and Accounting infrastructure for

marine data users (SeaDataNet, EMODNET, …).

12


Connecting B2ACCESS - Marine-ID

Marine-ID has been registered as an Identity Provider in

B2ACCESS allowing users to access Services connected to

B2ACCESS using their Marine-ID accounts.

13


Step 1: Demo page

Demo: https://snf-761524.vm.okeanos.grnet.gr/b2access

14


Step 2: Insert Credentials

15


Step 3: Marine-ID Consent Form

16


Step 4: B2ACCESS Consent form

17


Step 5: Yes you have logged in

18


Access to SDC web-based services

1. User clicks on the “login” button in the portal of a SeaDataCloud service

and he/she is redirected to B2ACCESS.

– Then B2ACCESS redirects the user automatically to Marine-ID.

2. The user enters his/her credentials.

3. The user is asked to consent the information that Marine-ID is going to

release to B2ACCESS.

– The user is redirected back to B2ACCESS.

4. If the user logins through B2ACCESS for the first time, he/she needs to

register to the platform. Otherwise, the user will be asked to consent the

information that B2ACCESS is going to release to the SeaDataCloud

service.

5. The user is redirected back to the SeaDataCloud service.

19


Register only the first time

20


More on WebODV integration - Sebastian

21


Quality control video


Data Extractor


ODV-online


REST API -- for developers (us) and advanced userscURL Request:curl -X POST "https://webodv.awi.de/testing/single_sample_extractor" -H "accept: applocation/json" -H "Content-Type: application/x-www-form-urlencoded" -d "dataset=SDN_test_Elba.odv&name=LOCAL_CDI_ID&value=8546&output=JSON"

JSON Response: {"station_id":642,"sample_id":1,"sample_count":379, …

...

,"datavars_id":[1,2,3],"datavars":["1.98","21.60","38.03"],"flags":["1","1","1"], …

…}

webODV REST API

(PHP)

Application

(PHP, JavaScript, …)


File selector

Client

File selector Service (API)

WebDAV


More on DIVA integration - Alexander,

Charles

29


DIVAnd in Jupyter notebooks● Jupyter notebooks are integrated web

environment− Computing− Visualization− Documentation of code

● DIVAnd extension of DIVA in more than 2 dimensions

● DIVAnd distributed as a Julia package installed into Docker containers


DIVAnd in Jupyter notebooks ● Loading bathymetry (GEBCO,

EMODnet)● Data formats (ODV spreadsheets,

netCDF, mat files etc.)● Support for databases like:

World Ocean DatabaseCMEMS INSTAC

● Check for duplicates● Automated quality checks ● Parameter optimisation● Error field computation● Plot results and data● Generate XML for catalog

Full example notebook


Integrations with SeaDataCloud services

● Querying the vocabulary service● Querying the EDMO database● XML files for Sextant● Extracting data from the ODV REST API● Embbed additional metadata for a given

observation based on EDMO code and local CDI id from jupyter notebook


REST API of DIVAnd● User provides

the URL of the data set (netCDF format)

● Provide analysis parameters and metadata

● Download the final analysis


Deployment-Pipeline

Source code repository

Code testing

Distribution of Docker images

on every commit to the master branch*

If all tests succeed

● Pipeline of the Docker images for the − Jupyter notebook − REST API

*Only fully implemented features are merged into the master branch


Biological QC – Simon

35


Additional visualisation – Giorgio

36


• Containers, Notebooks, Package, Visualization

• Dockerfile https://github.com/openearth/sdc-visualization

• Data type. NetCDF / tarfile from B2DROP

• Logging, import sdc_visualization


Docker image updates

• Custom logging format (both for notebooks and python code)

• Updates published to hub.docker.com/openearth/sdc-visualization

• Closer follows docker guidelines

• Some help needed with limiting restrictions needed (--privileged --cap-add SYS_ADMIN –device)


Mapbox application

• Interactive API

• Code is available @ hub.docker.com/openearth/sdc-visualization

• Reads NetCDF and subset based on time

• Some help needed with limiting restrictions (--privileged --cap-add SYS_ADMIN –device)


Movie example

40


Integration layer mockups (UI) - Peter

41

[email protected] – www.seadatanet.org 42

SeaDataCloud Technical Task Group meeting, Espoo, 10-11.9.18


Private workspace


3. VRE deadlines and deliverables

47


Where are we now?

• Development kick-off At Deltares

• Virtual code sprint in June 2018

• Physical workshops in Sopot and Helsinki

• Currently in third development cycle of 4 months

(Nov 2018 - Feb 2019)

• Two weekly telco’s / trello boards with actions

• Currently in integration phase:

– Many concepts already tested

– Basic functions working in the cloud

– Now development of dashboard for first user group

48


Project month Component remark

PM12 VRE specification document This document

PM15 Start actual developments VRE Kickoff to be decided

PM18 D10.3: Specification of sub-setting application anddevelopment plan

Led by IFREMER

PM18 D10.5: Specification of Ocean Data View online anddevelopment plan

Led by AWI

PM18 D10.7: Specification of DIVA online and development plan Led by ULiege

PM18 D10.9: Specification of Biology Data QC online anddevelopment plan

Led by VLIZ

PM18 D10.13: Specification of visualisation services and developmentplan

Led by Deltares

PM18 D10.19: Specification of MySeaDataCloud and developmentplan

Led by EUDAT

PM24 D10.10: Phase 1 of Biology Data QC online operational Led by VLIZ

PM24 D10.17: Specification of SOS viewing services and developmentplan

Led by 52N

PM26 D10.15: Specification of Oceanotron services and development plan Led by IFREMER

PM30 D10.2 First version Virtual Research Environment (VRE) in the cloudoperational

For internal users, related to selected use cases

PM30 D10.4: Sub-setting application operational in VRE Led by IFREMER

PM30 D10.6: Ocean Data View online operational in VRE Led by AWI

PM30 D10.8: DIVA online operational in VRE Led by ULiege

PM30 D10.14: Visualisation services operational in VRE Led by Deltares

PM30 D10.20: MySeaDataCloud operational Led by EUDAT

PM31 D10.18: SOS viewing services for data streams operational Led by 52N


PM40 D10.16: Upgraded Oceanotron services operational for SeaDataNetWP11 data products

Led by IFREMER


PM42 VRE operational for internal (and some public?) users New: No official deliverable49


Questions, or suggestions?

50

wp10 –virtual research environment – progress of vre ......• virtual code sprint in june 2018...

Documents