earth system grid (esg) - dkrz · myscript.sh) run with: $ bash myscript.sh or: $ chmod +x...

Earth System Grid (ESG)

24.02.2012 Estanislao Gonzalez MPI/DKRZ

Usage – a brief introduction

• What is ESG? – ESG@DKRZ

• Creating an Account

• Logging in

• Finding Data/Metadata

• ESG Security – CMIP5

• Downloading Data

• Getting Help

Summary

24.02.2012 2 Estanislao Gonzalez MPI/DKRZ

raw model output (temporary)

/work/ik0555/cmip5/… …/experiments/outdata/…

Data Flow and Archives

• System for managing data from environmental sciences

• Previous version: system serving CMIP3 data @PCMDI (size 36TB) – still running

• Now: federation of systems serving CMIP5 data (Current archive size 940TB, expected 2PB)

• MPI data in ESG 60TB (currently ~26TB) All of it has gone through QC.

• All ESG CMIP5 data is freely accessible to researchers. Some even to all, e.g. MPI data

What is ESG?


What is ESG?


• All Gateways have access to all the metadata i.e. search results are the same across the federation

• All Gateways trust users from other Gateways i.e. only one account required federation-wide

• Each Gateway manages their own data nodes i.e. expect to jump from one to others

• Data is gathered directly from the data nodes i.e. the download URL will be different from that of the Gateway and could be situated in different continents.

What is ESG?


What is ESG? – ESG@DKRZ


Full Text Search

Faceted Search

Help Links

Other Gateways

Summary

Info Area

Navigation Bar

http://ipcc-ar5.dkrz.de

Creating an Account


Creating an Account


https://ipcc-ar5.dkrz.de/myopenid/egonzalez

Creating an Account


Logging in


Logging in


This means you haven’t follow the link in the confirmation mail. Check your Spam folder!

Finding Data


Finding Data


File Selection

Dataset Title

Finding Data


DRS - Dataset ID


cmip5.output2.MPI-M.MPI-ESM-LR.rcp45.yr.ocnBgchem.Oyr.r1i1p

<project>.<product>.<institution>.<model>. <experiment>.<realm>.<cmor-table>.<ensemble member>

Finding Data


Finding Metadata (CIM)


ESG Security – CMIP5




At this time CMIP5 groups memberships can only be viewed at PCMDI Gateway!

Downloading Data


Downloading Data


File sub-selection

Downloading Data


Downloading Data


To retain the version number it’s better to store the wget script or the dml file!

It’s a Bash script Store as text (e.g. myscript.sh) Run with: $ bash myscript.sh Or: $ chmod +x myscript.sh $ ./myscript.sh NEVER: sh myscript.sh (it’s not an sh script)

It’s a list that can be used with the DML (Data Mover Lite)

It’s a download manager, requires a GUI and at this time it will only be displayed if there are GridFTP endpoints to download from.

Certificates are valid for 72Hs

Downloading Data - alternative


https://verc.enes.org/data/direct-data-access/data/query

• Security status is stored in:

• If accessing via browser – cookie

• If accessing via script – X509 Certificate

•CMIP5 memberships are stored centrally at PCMDI

•It’s common to be moved from one Gateway to another. You might have to input your OpenID more than once, though never your password.

ESG Security


• To get a X509 Certificate: • Download the MyProxyLogon app (needs a gui):

http://ipcc-ar5.dkrz.de/webstart/myProxyLogon/MyProxyLogon-ESG.jar

• Run using the proper parameters as defined in: http://ipcc-ar5.dkrz.de/help/download-help.htm

• Retrieved certificate will be in $HOME/.esg • In case of troubles, just wipe $HOME/.esg and

start anew.

ESG Security










http://ipcc-ar5.dkrz.de/help/download-help.htm






• The IS-ENES vERC page: https://verc.enes.org

• Downloading from CMIP5 Gateways: https://verc.enes.org/help/how-to-./esgf-gateways

• IS-ENES Search Service: https://verc.enes.org/help/how-to-./the-is-enes-data-search

Getting Help


https://verc.enes.org/

https://verc.enes.org/


https://verc.enes.org/help/how-to-./esgf-gateways









https://verc.enes.org/help/how-to-./the-is-enes-data-search















http://cmip-pcmdi.llnl.gov/cmip5/

• Getting Started Guide: http://cmip-pcmdi.llnl.gov/cmip5/data_getting_started.html?submenuheader=3

• Everything about downloading data: http://ipcc-ar5.dkrz.de/help/download-help.htm

• Data availability: http://cmip-pcmdi.llnl.gov/cmip5/availability.html

• General Info on CMIP5:http://cmip-pcmdi.llnl.gov/cmip5/

Getting Help

























• CMIP5 Helpdesk:

• [email protected]

Getting Help


mailto:[email protected]




Thank you.


earth system grid (esg) - dkrz · myscript.sh) run with: $ bash myscript.sh or: $ chmod +x...

Documents