data storage system, model output and analysis tools precis workshop tanzania meteorological agency...

Data Storage System, Model Output and Analysis ToolsPRECIS Workshop

Tanzania Meteorological Agency

29th June – 3rd July 2015

© Crown copyright Met Office

• In Linux, variables are identified with a preceding $. So $x, $TEMP, $David, $R2D2, $EXETER could all be variable names in Linux. Variables can hold text or numbers, including file names and directories.

• $HOME is a predefined Linux variable which refers to the main working directory of the user’s account. In Linux, this is usually specified by /home/[username] – e.g. $HOME=/home/precis

• The text file setvars is created at installation. It contains a long list of variable declarations which tell PRECIS where input and output data lives.

• Find setvars in $HOME/setvars (e.g. /home/precis/setvars)

Structure


• PRECIS write its output data to the directory set in setvars as $ARCHIVEDIR (/home/precis/archive by default)

• Each experiment is uniquely identified by a five character RUNID. Within $ARCHIVEDIR, there are subfolders according to the RUNID.

• Examples:

/home/precis/archive/abcab/

/home/precis/archive/exete/

/data/archive/jedaa (if ARCHIVEDIR=/data/archive)

Location of the data


ARCHIVEDIR in setvars


Inside $ARCHIVEDIR :

The STASH code

• Each variable type (temperature, precipitation, etc.) from the model is written to a separate directory defined by that variable’s 5-digit ‘STASH’ code

• $ARCHIVEDIR/RUNID/03236/

• 03236 = Mean temperature at 1.5m • $ARCHIVEDIR/RUNID/08208/

• 08208 = Soil Moisture Content

• $ARCHIVEDIR/RUNID/05216/

• 05216 = Mean total precipitation

• For a full list, see Appendix C of the PRECIS technical manual (www.precisrcm.com/tech_man.pdf)

http://www.precisrcm.com/tech_man.pdf


Inside $ARCHIVEDIR/RUNID:

• Files follow this general pattern:

• [RUNID]a.[file type][Date].[STASH code].[format]

• Example:

• abdaba.pji9140.03236.pp

• All file types are listed in Appendix D of the technical manual

Output file naming convention


Inside $ARCHIVEDIR/RUNID/03236

• The PRECIS models output data in the PP binary data format. This is the Met Office’s own format.

• Conversion tools are provided to convert PP data to

• CF compliant NetCDF (standard)

• GRIB (for GrADs and Ferret)

• ArcInfo ascii (e.g. for GIS/spreadsheets)

• See Section 6.2 of the technical manual for further discussion.

Output Data format

Analysis Utilities

• To use the output data in any way, you will need to make use of some kind of data analysis software utility.

• In general, you need to process your data and/or visualise it.

• Two tools are used during the PRECIS training workshop: CDO and NCL, details of which follow.

• Other analysis tools exist, such as the Met Office’s IRIS system.

Analysis Utilities (cont)

• The utilities GrADs, Ferret and CDAT are included on the PRECIS installation software, but no training in the use of these is given nor will any technical support in their use be provided!!!

• There is a suite of data analysis tools for PP data which are installed as part of PRECIS and are fully supported by the PRECIS team. You will use some of these tools during the workshop.

Details of output data conventions

• Reduces number of characters used in setting date

• First digit indicates the decade: [0-9a-z]

• Actually number of decades since 1800. E.g.:

• 0 1800-1810

• a 1900-1910

• g 1960-1970

• r 2070-2080

• Second digit indicate year in decade [0-9]

• Third digit represents month: [1-9a-c]

• Fourth digit is the day of the month: [1-9a-v]

• Fifth digit is the hour of the day: [0-9a-o]

The UM Date Stamp

• r4c10

• 0:00z 1st December 2074

• h93ab

• 11:00z 10th March 1979

• j3jun

• June 1993

• l56ud

• 13:00 30th June 2015

Some examples

• PRECIS outputs monthly and seasonal means by default

• You can also output daily mean and hourly mean output, but beware filling up the hard drive!

• GCMs use either a 360 day calendar (30 days per month) or the standard Gregorian calendar. PRECIS keeps the calendar of the driving GCM.

• With 360 day calendar, you have control over defining the seasons (e.g. If you want three four-month seasons per year, or two six month seasons, etc)

Temporal Frequency of Output

Data analysis and visualisation utilities useful for PRECIS

CDO and NCLwhat are they?

CDO and NCL are tools for data analysis and visualisation.

Different tools are useful for different activities e.g. CDO tools for analysis and NCL for visualisation.

In the worksheets these tools are used together to process, analyse and visualise PRECIS output.

CDO – Climate Data Operators

Developed by the Max Planck Institute (Germany) for use with GRIB 1 / GRIB2, NetCDF3 / NetCDF4 and other data formats.

Designed specifically for climate and NWP data analysis, there are more than 600 operators available.

Can be run on Linux, Windows, MacOS and others.

CDO examples• Calculate the JJA mean:

cdo selmon,6/8 infile.nc jja.nc

cdo timmean jja.nc jja.mean.nc

or, pipe the operators together using ‘-’

cdo timmean –selmon,6/8 infile.nc jja.mean.nc

• Re-grid data onto a target grid:

cdo griddes target_grid.nc > mygrid

cdo remapbil.mygrid file.nc regridded_file.nc

Free to download and documentation and support forums can be found at https://code.zmaw.de/projects/cdo

CDO – Climate Data Operators

https://code.zmaw.de/projects/cdo

NCL – NCAR Command Language (1)

• Developed by NCAR (National Center for Atmospheric Research, USA ) for use with GRIB 1/2, netCDF 3/4 and ascii data.

• Free interpreted language designed for scientific data processing and visualization.

• Can be run on Linux, Windows, MacOS and others.

• Can be typed directly into the NCL command line (very time consuming), or scripted.

• Produces plots of publishable quality.


Extensive documentation and hundreds of example scripts and plots at http://www.ncl.ucar.edu/index.shtml

http://www.ncl.ucar.edu/index.shtml


Xconv – for quick looks

• The xconv graphical utility (written by Jeff Cole at Reading University, UK) allows for quick visualisation of PP and NetCDF format files.

• xconv is limited in that it:

• Only creates box plots

• Does not display map outlines

• Allows no control over scale or colours used in plots

• Therefore, xconv is not a good tool for creating plots for reports and published works. It is best used as a tool to take a “quick look” at data.


Xconv

Main

Xconv

Window


Xconv

A sample

plot of

Soil

Moisture

Content

with

Xconv


Panoply

• Panoply is a netCDF, HDF and GRIB data viewer written by NASA.

• Panoply is written in Java. It runs on Windows, MacOS or Linux.

• Panoply plots data from NetCDF, HDF and GRIB data sets.

• You can:

• Slice and plot specific latitude-longitude, latitude-vertical, or time-latitude arrays from larger multidimensional variables.

• Overlay continent outlines or masks on lon-lat plots

• Change the scale and colour table

• Save plots to disk GIF, JPEG, PNG or TIFF bitmap images or as PDF or PostScript graphics files.


Panoply

The main window


A sample

plot of

precipitation

using

Panoply

Using tools together

The worksheets combine these tools to post process and analysis PRECIS data.

1. Use PRECIS tools to do initial post processing

2. Convert the data to NetCDF format

3. Use CDO to analyse the NetCDF data.

4. Use NCL to visualise (and process) the NetCDF data.

Questions

• Models with the Gregorian calendar (ERA) have only monthly, and 3-month seasonal available.

• The Base Date for climate meaning sets the reference date for the beginning of means.

• The ‘infinite time series’ of all climate means will coincide with this date.

• Example: If the base date is set to 01 Apr 1960, four month seasonal means will be apr/may/jun/jul, aug/sep/oct/nov, and dec/jan/feb/mar. Annual means will be represented by 01 Apr – 30 Mar.

• Take the initial spin-up year into consideration when choosing the base date.

“Climate Mean” Output (reference)

• For models with a 360 day calendar (such as those driving the HadAM3P GCM) up to four climate meaning periods may be set

• These allow for long timescale means to be calculated within the model (rather than by the user)

• Choice of means is determined by a comma separated list representing shortest to longest

• Meaning periods are nested, and each one is specified by how many multiples of the previous (shorter) period it is.



• Example 1: 30,3,4,10

• 30 : 30 day (monthly) means

• 3 : 30 * 3 = 90 day (seasonal) means

• 4 : 90 * 4 = 360 day (annual) means

• 10 : 360 * 10 = 10 year (decadal) means


• Example 2: 30,4,3,5

• 30: 30 day (monthly) means

• 4 : 30 * 4 = 120 day (4-month seasonal) means

• 3 : 120 * 3 = 360 day (annual) means

• 5 : 360 * 5 = 5 year (decadal) means

data storage system, model output and analysis tools precis workshop tanzania meteorological agency...

Documents