earth system grid (esg) - dkrz · myscript.sh) run with: $ bash myscript.sh or: $ chmod +x...

Earth System Grid (ESG)

24.02.2012 Estanislao Gonzalez MPI/DKRZ

Usage – a brief introduction

• What is ESG? – ESG@DKRZ

• Creating an Account

• Logging in

• Finding Data/Metadata

• ESG Security – CMIP5

• Downloading Data

• Getting Help

Summary

24.02.2012 2 Estanislao Gonzalez MPI/DKRZ

raw model output (temporary)

/work/ik0555/cmip5/… …/experiments/outdata/…

Data Flow and Archives

• System for managing data from environmental sciences

• Previous version: system serving CMIP3 data @PCMDI (size 36TB) – still running

• Now: federation of systems serving CMIP5 data (Current archive size 940TB, expected 2PB)

• MPI data in ESG 60TB (currently ~26TB) All of it has gone through QC.

• All ESG CMIP5 data is freely accessible to researchers. Some even to all, e.g. MPI data

What is ESG?

24.02.2012 4 Estanislao Gonzalez MPI/DKRZ

What is ESG?

24.02.2012 5 Estanislao Gonzalez MPI/DKRZ

• All Gateways have access to all the metadata i.e. search results are the same across the federation

• All Gateways trust users from other Gateways i.e. only one account required federation-wide

• Each Gateway manages their own data nodes i.e. expect to jump from one to others

• Data is gathered directly from the data nodes i.e. the download URL will be different from that of the Gateway and could be situated in different continents.

What is ESG?

24.02.2012 6 Estanislao Gonzalez MPI/DKRZ

What is ESG? – ESG@DKRZ

24.02.2012 7 Estanislao Gonzalez MPI/DKRZ

Full Text Search

Faceted Search

Help Links

Other Gateways

Summary

Info Area

Navigation Bar

http://ipcc-ar5.dkrz.de

Creating an Account

24.02.2012 8 Estanislao Gonzalez MPI/DKRZ

Creating an Account

24.02.2012 9 Estanislao Gonzalez MPI/DKRZ

Creating an Account

24.02.2012 10 Estanislao Gonzalez MPI/DKRZ

Creating an Account

24.02.2012 11 Estanislao Gonzalez MPI/DKRZ

https://ipcc-ar5.dkrz.de/myopenid/egonzalez

Creating an Account

24.02.2012 12 Estanislao Gonzalez MPI/DKRZ

Logging in

24.02.2012 13 Estanislao Gonzalez MPI/DKRZ

Logging in

24.02.2012 14 Estanislao Gonzalez MPI/DKRZ

Logging in

24.02.2012 15 Estanislao Gonzalez MPI/DKRZ

This means you haven’t follow the link in the confirmation mail. Check your Spam folder!

Finding Data

24.02.2012 16 Estanislao Gonzalez MPI/DKRZ

Finding Data

24.02.2012 17 Estanislao Gonzalez MPI/DKRZ

Finding Data

24.02.2012 18 Estanislao Gonzalez MPI/DKRZ

File Selection

Dataset Title

Finding Data

24.02.2012 19 Estanislao Gonzalez MPI/DKRZ

DRS - Dataset ID

24.02.2012 20 Estanislao Gonzalez MPI/DKRZ

cmip5.output2.MPI-M.MPI-ESM-LR.rcp45.yr.ocnBgchem.Oyr.r1i1p

<project>.<product>.<institution>.<model>. <experiment>.<realm>.<cmor-table>.<ensemble member>

Finding Data

24.02.2012 21 Estanislao Gonzalez MPI/DKRZ

Finding Data

24.02.2012 22 Estanislao Gonzalez MPI/DKRZ

Finding Data

24.02.2012 23 Estanislao Gonzalez MPI/DKRZ

Finding Data

24.02.2012 24 Estanislao Gonzalez MPI/DKRZ

Finding Metadata (CIM)

24.02.2012 25 Estanislao Gonzalez MPI/DKRZ

Finding Metadata (CIM)

24.02.2012 26 Estanislao Gonzalez MPI/DKRZ

Finding Metadata (CIM)

24.02.2012 27 Estanislao Gonzalez MPI/DKRZ

Finding Metadata (CIM)

24.02.2012 28 Estanislao Gonzalez MPI/DKRZ

ESG Security – CMIP5

24.02.2012 29 Estanislao Gonzalez MPI/DKRZ

ESG Security – CMIP5

24.02.2012 30 Estanislao Gonzalez MPI/DKRZ

ESG Security – CMIP5

24.02.2012 31 Estanislao Gonzalez MPI/DKRZ

ESG Security – CMIP5

24.02.2012 32 Estanislao Gonzalez MPI/DKRZ

ESG Security – CMIP5

24.02.2012 33 Estanislao Gonzalez MPI/DKRZ

ESG Security – CMIP5

24.02.2012 34 Estanislao Gonzalez MPI/DKRZ

ESG Security – CMIP5

24.02.2012 35 Estanislao Gonzalez MPI/DKRZ

At this time CMIP5 groups memberships can only be viewed at PCMDI Gateway!

Downloading Data

24.02.2012 36 Estanislao Gonzalez MPI/DKRZ

Downloading Data

24.02.2012 37 Estanislao Gonzalez MPI/DKRZ

File sub-selection

Downloading Data

24.02.2012 38 Estanislao Gonzalez MPI/DKRZ

Downloading Data

24.02.2012 39 Estanislao Gonzalez MPI/DKRZ

To retain the version number it’s better to store the wget script or the dml file!

It’s a Bash script Store as text (e.g. myscript.sh) Run with: $ bash myscript.sh Or: $ chmod +x myscript.sh $ ./myscript.sh NEVER: sh myscript.sh (it’s not an sh script)

It’s a list that can be used with the DML (Data Mover Lite)

It’s a download manager, requires a GUI and at this time it will only be displayed if there are GridFTP endpoints to download from.

Certificates are valid for 72Hs

Downloading Data - alternative

24.02.2012 40 Estanislao Gonzalez MPI/DKRZ

https://verc.enes.org/data/direct-data-access/data/query

• Security status is stored in:

• If accessing via browser – cookie

• If accessing via script – X509 Certificate

•CMIP5 memberships are stored centrally at PCMDI

•It’s common to be moved from one Gateway to another. You might have to input your OpenID more than once, though never your password.

ESG Security

24.02.2012 41 Estanislao Gonzalez MPI/DKRZ

• To get a X509 Certificate: • Download the MyProxyLogon app (needs a gui):

http://ipcc-ar5.dkrz.de/webstart/myProxyLogon/MyProxyLogon-ESG.jar

• Run using the proper parameters as defined in: http://ipcc-ar5.dkrz.de/help/download-help.htm

• Retrieved certificate will be in $HOME/.esg • In case of troubles, just wipe $HOME/.esg and

start anew.

ESG Security

24.02.2012 42 Estanislao Gonzalez MPI/DKRZ

• The IS-ENES vERC page: https://verc.enes.org

• Downloading from CMIP5 Gateways: https://verc.enes.org/help/how-to-./esgf-gateways

• IS-ENES Search Service: https://verc.enes.org/help/how-to-./the-is-enes-data-search

Getting Help

24.02.2012 43 Estanislao Gonzalez MPI/DKRZ

• Getting Started Guide: http://cmip-pcmdi.llnl.gov/cmip5/data_getting_started.html?submenuheader=3

• Everything about downloading data: http://ipcc-ar5.dkrz.de/help/download-help.htm

• Data availability: http://cmip-pcmdi.llnl.gov/cmip5/availability.html

• General Info on CMIP5:http://cmip-pcmdi.llnl.gov/cmip5/

Getting Help

24.02.2012 44 Estanislao Gonzalez MPI/DKRZ

• CMIP5 Helpdesk:

• cmip5-helpdesk@stfc.ac.uk

Getting Help

24.02.2012 45 Estanislao Gonzalez MPI/DKRZ

Thank you.

24.02.2012 46 Estanislao Gonzalez MPI/DKRZ