icos carbon portal progress september 2016bin... · • integration with neon (ringo) and africa...

ICOS Carbon Portal

Progress September 2016

Alex Vermeulen & ICOS CP team

Carbon Portal “up front”

A website offering users

• Search for data

• Quick view of datasets

• Download of data

• Visualization tools (spatial & temporal)

• Statistics tools

• Synthesis reports

• Educational materials

• Usage summaries

• Community platform for elaborated products

http://www.icos-cp.eu

Carbon Portal principles

• Interoperability

• Internal across themes (air, ecosystem, ocean)

• Other (EU) infrastructures

• Global domain

• Curation, Provenance, Transparancy

• Open linked data

• Scalable and open solutions

• Web standards

• Involvement of and interaction with users

https://github.com/ICOS-Carbon-Portal

ICOS ERIC Carbon Portal, Lund, Sweden | www.icos-cp.eu 3

https://github.com/ICOS-Carbon-Portal

ICOS Data

• Level 0 – raw sensor output (either mV or physical units)

• Level 1/NRT – calibrated and automatically assured data

• Level 2– final observation data products

• Level 3– elaborated data products, ICOS data

4ICOS ERIC Carbon Portal, Lund, Sweden | www.icos-cp.eu

ICOS Data Policy

• Open and free of charge access to ICOS data products

• License: Creative Commons 4.0 BY

• Attribution required, ICOS requests:– Good scientific practice of use

– Citation needed, ICOS provides suggestion

– Acknowledgement or co-authorship when reasonable

• Redistribution is allowed but not necessary:– ICOS provides free, easy and most up to date access to all data!

– No need for own storage, ICOS provides the best and easiest!

• Users are encouraged to contribute their products and

distribute through ICOS CP through CC 4 BY


Identification “strategy”

• A “complete” PID & metadata strategy is useful, from

data generation to results publication

• Some initial “cost” (time & effort), but increases

interoperability and reproducibility

• Parallel use of ePIC PIDs and DOIs

– Low-level ePIC for “mass data” & “work flow”; useful in PID collections

– DataCite DOIs for fully citable data with strong preservation policies;

mostly useful at “end” of data life cycle

U.

Sch

ward

mann,

Euro

pean P

ers

iste

nt

Identi

fier

Conso

rtiu

m (

ePIC

), 2

015


ICOS and EUDAT common services

Measurement stations

(National networks)

ICOS Carbon Portal

User 1 User 2 User 3

Ecosystem

Thematic

Centre

Atmospheric

Thematic

Centre

Oceanic

Thematic

Centre

Standardized

processing, quality

assurance & control

ICOS

repository

(data,

metadata)

Sensor data

• Data ingestion

• Metadata services

• Data discovery & access

• Usage tracking

• Data management

• Repository

administration

• Preservation planning

• User community support

Diverse user

communities, including

data producers and

other portals

High per-

formance

computing

services

Finalized

data productsMetadata

registry &

catalogue

services

B2SAFE

B2STAGE

PID

PID, AAI

B2FIND

PID

AAI

ICOS data flow, common services


Calibration

Labs

Ingestion

Station TCCP

(passthrough)


StationCP

(passthrough)TC CP Users

NRT;L2L0

L0 L0

B2SAFE

B2SAFE

PID

PID

Users CP Users

B2SAFE

B2SAFE

L3 L3

CC4

BY?

Lice

nse?

CPPID

CP

PID

PID

PID

Progress• Central is Data Upload Service

• Connected to ontology driven persistent versioned metadata store

• Tasks

• Registers metadata of all data objects

• Mints PID’s (ePIC)

• Registers and receives checksummed data objects

• Replicates object to trusted repository (B2SAFE)

• Enriches metadata with ICOS specific ‘static metadata’ for landing page,

minimum Inspire/ISO19115

• Creates (dynamic) landing page/PID end point including evt. citation

• Facilitates machine-machine interaction

• Integrates with AAI (voluntary user registration)

• License information in metadata and landing page

• Enables usage tracking through PID

• Enables attribution/citation through PID

• Meta data registered at other (e.g. GEOSS) portals

• Assures data integrity and transparency

• Supports RDA recommendations for citation of dynamic datasets and

collections: query PID

Motto: Make use simple and render redistribution obsolete and completely useless!


Progress…

• Data upload tool for L0, L2 and L3 using REST API

• L0 upload for ETC data

• First two L3 datasets published at CP

• Scalable cloud services, VRE’s (EGI <->EUDAT):– Cloud service tests for footprint provision for atmosphere

– Modeller interactive interface using Jupyter python notebooks:

• Global Carbon project

• Theme 2 ENVRI+: data identification and citation, Use Case (ICOS) and several implementation cases

• Integration with NEON (RINGO) and Africa network (SEACRIFOG)

• CP as central repository for Inversion systems (CP/Eurocom workshop Lund, June 2016)

• Provision of pre-ICOS data (Obspack, link to WMO GAW and IG3IS)


Danger

https://data.icos-cp.eu/stilt/

Metadata editor: ICOS Station labelling Service


ICOS Data Service https://data.icos-cp.eu/portal/#search


https://data.icos-cp.eu/portal/#search

ICOS Data Service


Example data object landing page


Data license page

https://www.icos-cp.eu/license


https://www.icos-cp.eu/license

Thank You!

Read more...

• Materials from DataCite & ePIC workshop “Persistent Identifiers: Enabling Services for Data Intensive Research” (Paris, 2015) – see https://blog.datacite.org/recap/

• Deliverable 5.1 from ENVRIplus (contact Alex or Maggie)

• Socha – “Out of cite, out of mind”, Data Science Journal 12 (2013) – see https://www.jstage.jst.go.jp/article/dsj/12/0/12_OSOM13-043/_pdf

• Research Data Alliance (RDA) work groups – see http://rd-alliance.org


https://blog.datacite.org/recap/

https://www.jstage.jst.go.jp/article/dsj/12/0/12_OSOM13-043/_pdf

http://rd-alliance.org/

icos carbon portal progress september 2016bin... · • integration with neon (ringo) and africa...

Documents