idc development lan operator's guide - atm - home: · pdf file ·...

IDC/DEVLAN/OG-ATM

20 May 2008

English only

AWST-TR-08/07

IDC Development LAN Operator's Guide - ATM

This document contains information on the operation of the Atmospheric Transport

Modelling Software on the IDC Development LAN, including baseline descriptions and

monitoring, troubleshooting and change procedures.

Summary

This document describes the procedures used to operate and maintain the Atmospheric Transport Modelling

software on the IDC Development LAN. It also describes the hardware, software and environment baseline,

which is needed to properly operate the Development LAN software. The chapters on the Development LAN

baseline and on change procedures are of general interest for all Development LAN users and developers.

IDC/DEVLAN/OG-ATM

May-08

Document history

Version Date Author Description

0.1 7 June 2004 Alexander Boresch draft document structure

0.2 12 August 2004 Marian Harustak partial draft procedures in chapters 3 and 4

0.3 16 September 2004 Alexander Boresch complete draft version

0.4 15 October 2004 Alexander Boresch revised draft version

1.0 28 November 2004 Alexander Boresch final SHI version

1.1 16 August 2005 Alexander Boresch baseline for expanded version (SHI, RN, ATM)

1.2 7 February 2006 Henrik Edlund ATM chapters added

1.3 14 December 2006 Gertrud Waich added autoconf build procedures, updated SHI

chapters

2.0 5 January 2007 Gertrud Waich,

Alexander Boresch

full SHI, RN and ATM version

3.0 20 May 2008 Vladimir Gelashvili,

Jan Wüster (review)

Created a separate volume for ATM software;

restructured the document to make it consistent with

SHI and RN volumes

IDC/DEVLAN/OG-ATM

May-08

Contents

1. Introduction ....................................................................................................................5

1.1. Identification and Purpose of the Document ............................................................5

1.2. Purpose and Role of the Development LAN............................................................5

1.3. Document Overview ...............................................................................................6

1.4. Typographical Conventions.....................................................................................6

1.5. Status of document release ......................................................................................6

2. General Baselines and Procedures for all Technologies...................................................7

3. ATM Processing.............................................................................................................7

3.1. Introduction ............................................................................................................7

3.2. Baseline inventory ..................................................................................................8

3.2.1 Hardware baseline ...........................................................................................8

3.2.2 Application software baseline..........................................................................8

3.2.3 Software location ..........................................................................................12

3.2.4 Software configuration baseline.....................................................................12

3.2.5 Development LAN ATM directory structure .................................................13

3.2.5.1 Top level ATM directories.........................................................................13

3.2.5.2 atm directory ............................................................................................13

3.2.5.3 bin directory ............................................................................................13

3.2.5.4 config directory .....................................................................................13

3.2.5.5 lib directory ............................................................................................14

3.2.6 Development LAN ATM user environment...................................................14

3.2.6.1 User environment for automatic processing ...............................................14

3.2.6.2 User environment for interactive processing ..............................................15

3.2.7 ATM data acquisition....................................................................................15

3.2.8 ATM automatic processing............................................................................16

3.2.9 ATM interactive processing...........................................................................16

3.2.10 ATM post-analysis processing.......................................................................16

3.3. Operations Procedures ..........................................................................................17

3.3.1 Starting the ATM processing system .............................................................17

3.3.2 Shut down of the ATM processing system.....................................................18

3.4. Monitoring Procedures..........................................................................................19

3.4.1 Routine monitoring tasks...............................................................................19

3.4.2 Layer 1 monitoring........................................................................................19




3.4.6 Log files ........................................................................................................19

3.4.7 Maintenance procedures ................................................................................19

3.5. Troubleshooting Procedures ..................................................................................20

3.5.1 Interpreting log and error messages ...............................................................20

3.5.2 Solving common problems ............................................................................21

3.5.3 Handling unknown problems.........................................................................21

Terminology.........................................................................................................................22

Glossary ...........................................................................................................................22

Abbreviations ...................................................................................................................25

References............................................................................................................................27

4. Desirable Enhancements (TODO-list)...........................................................................29

IDC/DEVLAN/OG-ATM

May-08

List of Figures

Figure 1. IDC technologies and data processing systems ........................................................5

Figure 2. ATM Software Control Flow Diagram...................................................................11

IDC/DEVLAN/OG-RN

May-08

1. INTRODUCTION

1.1. Identification and Purpose of the Document

This document describes the procedures used to operate and maintain the Atmospheric

Transport Modelling (ATM) software version 2.0 on the IDC Development LAN. It provides

an overview of the existing infrastructure and also describes the hardware, software and

environment baseline, which is needed to properly operate the Development LAN software.

The addressed audience is primarily the Development LAN operators, but also includes all

developers, testers, users and administrators of the Development LAN hardware and software.

There are two companion documents, [IDC/DEVLAN/OG-SHI] and [IDC/DEVLAN/OG-

RN], which describe the Development LAN’s Operations and Maintenance procedures for the

waveform technologies processing software and the radionuclide data software, respectively.

The current document is based on general framework provided by the following documents:

• IDC Operational Manual, [IDC-OM]: Describes the general operational framework and

mission of the IDC. Available at H:\Conference Documents\Official Documents\PMOs

and AG\WGA\WGB\Operational Manuals\IDC OM in the CTBTO Intranet.

• Draft Procedures of the IDC Configuration Control Board (CCB), [IDC-CCB]: Describes

the software and configuration change procedures for the IDC Testbed and Operations

LANs. Available at: http://idc030.idc.ctbto.org:8000/Docs/CCB-Documentation.html in

the CTBTO Intranet.

IDC Software Documentation Orientation Booklet, [IDC-SDO]: Introduces new users to the

IDC work environment including the IDC computer infrastructure and software. It is available

at http://intranet.ctbto.org/stafforientation02mar20.pdf .

1.2. Purpose and Role of the Development LAN

The Development LAN was created in July 2002 and serves as a development and test

platform for the IDC software. This includes processing software for all CTBT monitoring

technologies Seismic, Hydro-acoustic and Infrasound (SHI) data processing, Radionuclide

(RN) data processing and Atmospheric Transport Modelling (ATM).

Figure 1. IDC technologies and data processing systems

The Development LAN is owned by the IDC Software Integration unit and is under less

rigorous configuration control than other IDC LANs (Testbed and Operations LANs). When

new or modified software becomes available and has successfully passed unit testing by the

IDC/DEVLAN/OG-ATM

May-08

developers, it is installed on the Development LAN to be tested in the integrated IDC data

processing system for the relevant technology. Such integration testing helps to identify any

unintended effects of the installed software change on other processing software. If such

effects are found, they are analyzed and resolved before the software change is implemented

on the Testbed LAN for operational testing, and finally on the Operations LAN for use in

regular IDC operations. Thus the Development LAN has a major quality assurance role in the

IDC software development cycle.

1.3. Document Overview

Chapter 1. provides an introduction to this document.

Chapter 2. contains general information, which is common for all processing technologies. It

is organized in sections describing roles and responsibilities, the hardware and infrastructure

baselines, the general directory structure, the role and use of the configuration management

system ClearCase at the IDC, and generic change procedures. The information in this chapter

is of general interest for all Development LAN users. Chapter 2. is identical for all three

volumes of the Development LAN Operator’s Guide. To support maintainability it is

only contained in [DEVLAN/OG-SHI] and omitted here.

Chapter 3. contains all ATM-specific information including the baseline inventory and

procedures for routine operations, system monitoring, and system maintenance and

troubleshooting. The primary audience for this chapter are the operators of the ATM

processing software. However, the baseline information and troubleshooting procedures may

also be of interest for developers and testers.

1.4. Typographical Conventions

Element Font Example

application, process,

programming language

italic WorkFlow, Perl

database account, database table,

database attribute

UPPERCASE.bold.normal IDCX.interval.intvlid

configuration parameter, variable,

literal value, command, computer

code, machine name

Courier New 11 ssh cmss@eldey,

$(CMS_CONFIG)

1.5. Status of document release

This ATM volume is released as a work in progress. It is expected to be updated as more

information becomes available and experience is gathered with ATM on the Development

LAN. Specific gaps in available information are marked with [TBD] (to be determined).

Information in need of review and possibly update and revision is marked with [TBV] (to be

verified). The last section of the document DESIRABLE ENHANCEMENTS contains a to-do

list.

IDC/DEVLAN/OG-RN

May-08

2. GENERAL BASELINES AND PROCEDURES FOR ALL TECHNOLOGIES

This chapter is identical to the three volumes of the Development LAN Operator’s Guide. To

facilitate document maintenance it is omitted here. See [IDC/DEVLAN/OG-SHI] to see the

contents.

3. ATM PROCESSING

3.1. Introduction

The Atmospheric Transport Modelling (ATM) software uses wind field data, in relation with

identification of specific nuclides at Radionuclide (RN) stations, to derive likely geographical

origins for potential releases of radionuclides into the earth’s atmosphere. It calculates various

types of Source-Receptor-Sensitivity (SRS) matrices, Fields of Regard (FOR) and Possible

Source Regions (PSR), to be visualized, attached to the Reviewed Radionuclide Report

(RRR) and further analyzed in connection with the Radionuclide (RN) products.

Since its first version the components of the ATM software have been grouped into four so-

called layers. Though currently this division does not play such an important role as before, it

provides a convenient logical framework to presenting the software’s functionality.

• Layer 1 is responsible for pre-processing activities: the retrieval of suitable

meteorological input data required for global dispersion modelling. There is a set of

applications that retrieves data from the European Centre for Medium-Range Weather

Forecasts (ECMWF), and a set of applications that retrieves data from the U.S.

National Centre for Environmental Prediction (NCEP).

• Layer 2 contains the software for SRS modelling, which links hypothetical source

regions to the likely destinations of radionuclides originating in the source regions by

modelling the dispersion of hypothetically released radionuclides under influence of

the observed wind fields.

• Layer 3 contains an application to convert the SRS matrices to FORs. [TBV]

• Layer 4 combines several applications to produce the ATM products and to provide

them to all authorized users via the IDC products Web site and to external subscribers

via the subscription subsystem.

The ATM software is required to operate both at the IDC and at National Data Centres

(NDCs). The current ATM 2.0 system, developed and supported by the Software Applications

Section of the IDC, replaces the ATM 1.0 system completely. ATM 2.0 was developed to

improve the portability and maintainability of the software and to bring it in compliance with

current IDC software development standards.

ATM 2.0 serves as the basis for an in-house Research&Development project at the CTBTO

Provisional Technical Secretariat (PTS) with the objective to develop a so-called Ensemble

Dispersion Modelling (EDM) system [B+07], which will make use of backward modelling.

IDC/DEVLAN/OG-ATM

May-08

3.2. Baseline inventory

3.2.1 Hardware baseline

ATM software runs on Linux Red Hat 5.1 based machines called dls012 and dls013.

Layer 3 and 4 applications can be also run on local user workstations.

3.2.2 Application software baseline

The central part of the ATM modelling is constituted by the FLEXPART and HYSPLIT

software packages, which perform the ATM modelling routines.

• FLEXPART is a comprehensive tool for atmospheric transport modeling and analysis

that uses the Lagrangian particle dispersion model FLEXPART.

• HYSPLIT stands for Hybrid Single-Particle Lagrangian Integrated Trajectory. It is a

complete system for computing trajectories of complex dispersion and deposition

simulations using either puff or particle approaches.

The FLEXPART and HYSPLIT software packages are in the public domain. They are not

described in more detail in the current guide. Please refer to [DSRT05] and [SS02] for

additional information.

Other software components were written at the IDC to encapsulate and control the modelling

packages and to perform data retrieval, pre- and postprocessing for them. Software portability

and maintainability were important requirements for this development, so these components

were written in the Perl programming language.

The ATM software Perl modules are described below in the current section. The terms

'module' and 'component' are used interchangeably. In the description use is made of a

standard convention of Perl package deployments: the “::” sign stands for a directory

separator and the last part of the component name becomes the file name with the extension

“.pm”.

For example,

Local::ATM::Acquisition stands for [...]/Local/ATM/Acquisition.pm.

The main Perl components of theATM processing software are:

Component Functionality Local::ATM::Acquisition This is the umbrella component of all

functionality related to acquiring data. It will

load the configured protocol component (FTP

or SFTP), create a protocol object and then

call the configured acquisition mode

component (Mirror or Schedule). Local::ATM::Acquisition::FTP Perl modules specific to data retrieval via

FTP.

IDC/DEVLAN/OG-RN

May-08

Component Functionality Local::ATM::Acquisition::Schedule This component handles acquisition of

remote files that become available on a

scheduled basis. Local::ATM::Acquisition::SFTP Perl module specific to data retrieval using

SFTP protocol Local::ATM::Acquisition::Mirror This component handles acquisition of

remote files and can use parts of the remote

name to create a local directory and a

filename. [TBV] Local::ATM::Modelling Perl module that handles common modelling

tasks. Local::ATM::Modelling::FLEXPART This component handles pre-modelling and

post-modelling tasks which are specific to the

FLEXPART model. Local::ATM::Modelling::HYSPLIT This component handles pre-modelling and

post-modelling tasks which are specific to the

HYSPLIT model. Local::ATM::Visualisation Perl module that handles visualisation tasks,

like the production of plots and the

production of the RRR attachment (plots,

web page and link to SRS field data file).

The following processing components provide functionality to launch or to support the above-

mentioned processing components:

Component Functionality atm Perl script that is run to launch the ATM 2.0

system Local::ATM This component acts as the catch-all for any

uncaught error that occurs in other (called)

components so that these can be properly

logged. It also logs the command line

arguments given when atm is called. Local::ATM::Dispatch Perl module implementing the dispatch

method, used to determine what major ATM

component (acquisition, modelling or

visualisation) to launch and then launch it. Local::ATM::Station The component encapsulates all information

about one station and makes this accessible

via various methods. Local::ATM::Sample The component encapsulates all information

about one sample and makes this accessible

via various methods. Local::ATM::Constants The component provides importable

constants for other components.

IDC/DEVLAN/OG-ATM

May-08

Component Functionality Local::ATM::Default The component provides functions which

determine default parameter values for other

components. Local::ATM::Preprocess The component provides functions for

converting raw configuration parameters

values (strings) to an internal representation

(Perl data). Local::ATM::Type The component provides several commonly

used functions which are used for parameter

validation by other components.

The following Perl modules constitute the interface components to the logging functionality,

database and various types of configuration and data files:

Component Functionality Local::ATM::Config The component acts as a superclass for

many other components and provides a

constructor (new) that will initialize an

object’s attributes from a configuration

source. Local::ATM::Config::Singleton The component loads configuration

from configuration files via command

line parameters Local::ATM::Config::Station The component provides a constructor

(new) that will initialize the object’s

attributes from a configuration source. Local::ATM::Config::Station::Singleton The component loads configuration

from configuration files specified via

the provided configuration attributes. Local::ATM::Config::Sample The component provides a constructor

that will initialize the object’s attributes

from a sample parameter source. Local::ATM::Config::Sample::Singleton The component loads configuration for

a sample via a default (station

configuration) or otherwise configured

sample configuration source. Local::ATM::Log The component acts as a superclass for


number of methods to be inherited for

logging. Local::ATM::Log::Base The component provides shared

methods for the logging components Local::ATM::Log::Email The component provides logging via

email Local::ATM::Log::File The component provides logging via

file

IDC/DEVLAN/OG-RN

May-08

Component Functionality Local::ATM::Config The component acts as a superclass for


constructor (new) that will initialize an

object’s attributes from a configuration

source. Local::ATM::Log::Screen The component provides logging via

screen (e.g., terminal). Local::ATM::Log::Syslog The component provides logging via

Syslog. Local::ATM::State The component acts as a superclass for


number of methods to be inherited for

state tracking. Local::ATM::State::Database This component provides state

reporting to a database.

A simplified general control flow diagram is displayed below in Figure 2

Figure 2. ATM Software Control Flow Diagram.

The diagram omits components of supportive nature (Local::ATM::Constants,

Local::ATM::Default,Local::ATM::Dispatch::Session,Local::ATM::Preproc

ess, Local::ATM::Station, Local::ATM::Sample and Local::ATM::Type), as well

as all interfaces (configuration, logging, state information) and all external components.

Considering the software division in layers described in the previous section, the mapping

between actual software modules and the layers is done in the following way:

• Layer 1 processing is handled mainly by Local::ATM::Acquisition (and

subcomponents);

• Layer 2 and Layer 3 functionality is organized mainly by

Local::ATM::Modelling module (and subcomponents) ; [TBV]

• Layer 4 processing is handled by Local::ATM::Visualisation;

IDC/DEVLAN/OG-ATM

May-08

Other software components fall outside the layer division or can be used by several layers.

All ATM software applications are located below the directory /dvl/software/atm. All

ATM specific Perl modules reside in /dvl/software/atm/lib. See section 3.2.5 for more

detailed information on the directory structure.

The automatic ATM processing software is run by the UNIX/Linux user atmops. The user

atmops is maintained by the Development LAN operator.

The automatic part of the software is controlled by cron jobs.

Like the other components of the IDC software, the ATM software is subject to configuration

management, using the IDC’s configuration management system ClearCase as a tool. The

Development LAN Operator maintains the application software in ClearCase and in the

Development LAN runtime system. The ClearCase administrator maintains the IDC software

baseline for other LANs and assigns the Development LAN software patch labels. Refer to

section 2.6 of the SHI part of the Operator's Guide for additional information on requesting

and implementing changes to the Development LAN application software.

The following ClearCase VOB contains the ATM software at the IDC:

/vobs/idc/external.

In case of the ATM software, only the core applications and all configuration data files are

under ClearCase control, while currently the Perl modules belonging to the ATM software

package reside only in the run-time system. This is expected to change in the near future.

3.2.3 Software location

All ATM software is located below the directory /dvl/software/atm.

/dvl/software/atm/bin contains symbolic links pointing to the binaries in the

subdirectory /dvl/software/atm/bin/Linux-x86_64-RHEL5/, which is specific to the

Operating System RedHat 5.1. The indirection to an OS-specific directory is introduced in

order to support the use of the mechanism for inhomogeneous data processing systems,

although the ATM software exists in the version for RH5.1 only. The use of isaexec.sh is

discussed in [IDC/DEVLAN/OG-SHI] section 3.2.5.2.

For more detailed information about the directory structure please check section 3.2.5 of the

current document.

3.2.4 Software configuration baseline

All configuration files for the ATM processing system are located in the directory

/dvl/software/atm/config. More detailed description on the directory content is given

in section 3.2.5.4.

IDC/DEVLAN/OG-RN

May-08

3.2.5 Development LAN ATM directory structure

The general top-level directory structure for the Development LAN is described in chapter 2. .

The ATM-specific top level directories follow this general structure and are listed in the table

below. Directory levels are counted from the top level (level 1), which is $(lan)=/dvl for

the Development LAN. In the ATM area, the subdirectory depth goes to 8.

3.2.5.1 Top level ATM directories

Top level (Level 3) ATM directories $(lan)/software/atm All ATM processing software. $(lan)/logs/atm ATM log file area $(lan)/products/atm ATM data products $(lan)/data/atm ATM data

3.2.5.2 atm directory

Level 4: ATM data processing directories $(lan)/software/atm All ATM processing software. ./bin ATM binaries and scripts [TBV] ./config ATM processing configuration files ./lib ATM Perl modules

3.2.5.3 bin directory

Level 5: ATM executables /dvl/software/atm/bin Contains symbolic links to ATM binaries and

scripts [TBV] of the corresponding OS. ./Linux-x86_64-RHEL5/ Contains executables for RedHat 5.1

3.2.5.4 config directory

$(lan)/software/atm/config/ is the upper level directory containing ATM, system

and station parameters. It contains the following sub-directories:

Level 5-7: ATM processing parameter configuration $(lan)/software/atm/config Contains ATM, logging, database and station

parameters. ./atm/ Root directory for ATM configuration files ./atm/acquisition Configuration files relevant to data

acquisition ./atm/log Configuration files relevant to various

logging functionality ./atm/modelling/flexpart FLEXPART configuration ./atm/modelling/hysplit HYSPLIT configuration ./atm/state Configuration files for the software that

handles state reporting for the entire ATM

system

IDC/DEVLAN/OG-ATM

May-08

Level 5-7: ATM processing parameter configuration ./atm/visualisation Visualisation software ./static/ Processing state reporting configuration for

the entire ATM system ./stations/ Station specific parameters ./system/ System specific parameters

3.2.5.5 lib directory

Level 5-8: ATM Perl modules $(lan)/software/atm/lib Contains all Perl modules relevant to the

ATM processing. ./perl Contains Local subdirectory ./Local Root directory for Perl modules ./Local/ATM Root directory for Perl modules specific to

ATM processing ./Local/ATM/Acquisition Perl modules specific to data acquisition ./Local/ATM/Config Perl modules specific to ./Local/ATM/Config/Sample Perl modules specific to sample

configuration ./Local/ATM/Config/Station Perl modules specific to station configuration ./Local/ATM/Dispatch Perl modules specific to dispatch [TBV] ./Local/ATM/Log Perl modules specific to logging activities ./Local/ATM/Modelling Perl modules specific to ATM modeling ./Local/ATM/State Perl modules specific to processing state

reporting

3.2.6 Development LAN ATM user environment

3.2.6.1 User environment for automatic processing

The following environment variables have to be set in order to operate the automatic ATM

processing software: [TBV]

GRIBLIB=TBD

METLIBHOME=TBD

ECMWF_LOCAL_TABLE_PATH=\

/dvl/software/site/.rhel5/usr/lib64/gribtables/

LOCAL_DEFINITION_TEMPLATES=\

/dvl/software/site/.rhel5/usr/lib64/gribtemplates/

GRIB_DEFINITION_PATH=\

/dvl/software/site/.rhel5/usr/share/grib_api/definitions

GRIB_TEMPLATES_PATH=\

/dvl/software/site/.rhel5/usr/share/grib_api/templates

IDC/DEVLAN/OG-RN

May-08

3.2.6.2 User environment for interactive processing

There is a standard ATM user environment based on csh, which should be used by all users

who want to develop, test or use any ATM software on the Development LAN. The

environment variables are defined in the file .cshrc in the home directory of the atmops

user. These variables are automatically read and used by the ATM system when run via cron

the next time.

The relevant environment variables are presented below in alphabetical order with their value

on the Development LAN, information in which software layers they are used and a short

description: [TBV]

Variable Value Layers Description ATMMODHOME /dvl/software/atm/\

atm-layer2/SRSM-MODEL 3, 4 home of the current

dispersion model software ATMOPERATOR "[email protected]" 1, 2, 4 development LAN specific,

used where possible instead

of hard-coded email

addresses in scripts ECMWFDATAHOME /dvl/software/atm/\

atm-layer1/ECMWFDATA

1, 2 home of the ECMWF

retrieval software layer 1 GRIBLIB /dvl/software/atm/\

atm-layer2/SRSM-MODEL/\

emosdir_000220

1, 2, 3 home for emos, required for

building software

METLIBHOME /dvl/software/atm/\

atm-layer2/SRSM-MODEL/\

metlib

1, 2, 3 home for metlib, required for

building software

NCARG_ROOT /usr/local/ncarg 4 home of the NCAR Graphics

software

NCEPDATAHOME /dvl/software/atm/\

atm-layer1

NCEPDATA

1, 2 home of the NCEP retrieval

software in layer 1

PATH $PATH:$NCARG_ROOT/bin 4 the binary directory for the

NCAR Graphics software is

included in the PATH

PLOTFORHOME /dvl/software/atm/\

atm-layer4/plotfor 4 home for the plotting

software in layer 4

POSTPROCHOME /dvl/software/atm/\

atm-layer3 3, 4 home for the post-processing

software in layer 3

3.2.7 ATM data acquisition

ATM data acquisition works by regularly checking availability and by pulling the most recent

wind-field data off specified, publicly available URLs at two meteorological institutions,

ECMWF and NCEP.

The csh script get-ecmwf.csh launches the data acquisition; it is under control of the cron

demon.

Wind field data is retrieved as 3- or 6-hourly snapshots. Retrieval takes place at configured

intervals based on crontab entries and parameter files depending on data availability. For

IDC/DEVLAN/OG-ATM

May-08

example NCEP GDAS consists of 4 data files per day (00, 06, 12, 18 hours analysed

snapshots).

3.2.8 ATM automatic processing

Layer 2 software (modelling) is run automatically under control of cron. It. consists of two

particle trajectory models (FLEXPART_5.1 [SS02]; HYSPLIT_4.7 [DSRT05]). Output of the

modelling is sample specific SRS fields. Modelling takes place backwards over 14 or 21 days

with 8, 12 or 24 hours of releases.

Layer 3 software (result conversion) is run automatically under control of cron. It provides

post-processing of modelling results, converting SRS fields into several Fields of Regard

(application SRS2FOR, processing part) and PSR products.

Layer 4 software (visualization) is partly run automatically under control of cron and partly

interactively on demand. The automatic part performs visualization of the layer 3 output in

batch mode (SRS2FOR, plotting part).

3.2.9 ATM interactive processing

Visualization can be performed interactively on demand using the application WEB-GRAPE.

The WEB-GRAPE stands for Web-Connected Graphics Engine. This software has been

developed by the PTS as an end-user tool to interpret the SRS fields provided by the IDC.

This software can be used both to calculate and to interpret graphically the FOR and PSR

products. It is not routinely used on the Development LAN.

For more details please refer to WEB-GRAPE documentation. [TBD]

3.2.10 ATM post-analysis processing

Not applicable to ATM.

IDC/DEVLAN/OG-RN

May-08

3.3. Operations Procedures

Until now the Development LAN operators have not gathered much experience with the ATM

software. This section will be enhanced and enlarged as more experience is gathered

3.3.1 Starting the ATM processing system

The ATM processing system is controlled by the cron jobs listed below. If one or several of

these have been disabled and the corresponding process is thus not being run, it can be

activated by removing the comment sign (crochet #) from the respective line in the crontab

file.

# Acquisition Schedule ECMWF

05,20,35,50 * * * * /dvl/software/atm/bin/atm \

par=/dvl/software/atm/config/atm/acquisition/schedule/ecmwf/\

ecmwf.par atm_session_id=1

# Modelling FLEXPART ECMWF

00,30 08-13 * * * /dvl/software/atm/bin/atm \

par=/dvl/software/atm/config/atm/modelling/flexpart/ecmwf/\

ecmwf.par atm_session_id=11 \

atm_modelling_stations=VIP00,ARP01,ARP02,ARP03,AUP04,\

AUP05.AUP06,AUP07,AUP08,AUP09,AUP10,BRP11,BRP12,CMP13




atm_modelling_stations=CAP14,CAP15,CAP16,CAP17,CLP18,\

CLP19,CNP20,CNP21,CNP22,CKP23,ECP24,ETP25,FJP26,FRP27




atm_modelling_stations=FRP28,FRP29,FRP30,FRP31,FRP32,\

DEP33,ISP34,IRP36,JPP37,JPP38,KIP39,KWP40,LYP41




atm_modelling_stations=MYP42,MRP43,MXP44,MNP45,NZP47,\

NEP48,NOP49,PAP50,PGP51,PHP52,PTP53,RUP54,RUP55




atm_modelling_stations=RUP56,RUP57,RUP58,RUP59,RUP60,\

RUP61,ZAP62,SEP63,TZP64,THP65,GBP66,GBP67,GBP68



IDC/DEVLAN/OG-ATM

May-08


atm_modelling_stations=USP70,USP71,USP72,USP73,USP74,\

USP75,USP76,USP77,USP78,USP79

# Visualisation FLEXPART


par=/dvl/software/atm/config/atm/visualisation/flexpart/\

flexpart.par atm_session_id=11 \

atm_modelling_stations=VIP00,ARP01,ARP02,ARP03,AUP04,\

AUP05,AUP06,AUP07,AUP08,AUP09,AUP10,BRP11,BRP12,CMP13




atm_modelling_stations=CAP14,CAP15,CAP16,CAP17,CLP18,\

CLP19,CNP20,CNP21,CNP22,CKP23,ECP24,ETP25,FJP26,FRP27




atm_modelling_stations=FRP28,FRP29,FRP30,FRP31,FRP32,\

DEP33,ISP34,IRP36,JPP37,JPP38,KIP39,KWP40,LYP41














atm_modelling_stations=USP70,USP71,USP72,USP73,USP74,\

USP75,USP76,USP77,USP78,USP79

3.3.2 Shut down of the ATM processing system

In order to disable the automatic processing it is sufficient to comment out the relevant

crontab entries on the processing machines.

IDC/DEVLAN/OG-RN

May-08

3.4. Monitoring Procedures



3.4.1 Routine monitoring tasks

The ATM processing software is able to report processing errors via various methods. These

are: log file, Syslog, screen (interactive in case of interactive processing) and email. All

debug, info, notice, warning and error messages are extensively being reported to all

configured destinations depending on the configured reporting minimum and maximum

levels.

3.4.2 Layer 1 monitoring

Layer 1 software is responsible for data retrieval. So the main monitoring task consists in

checking that configured wind field data is successfully being retrieved. This is done by

regularly observing the contents of the /dvl/data/atm directory for all configured

acquisitions. Also WorkFlow can be used to spot processing failures.


Monitor that all configured modelling is running [TBD, how] and producing SRS products in

/dvl/products/atm. WorkFlow also provides useful information for this software layer.


[TBD]


Monitor that all configured visualizations are running and produce RRR attachments to the

Reviewed Radionuclide Report (RRR) webpage using the SRS data products from layer 2.

Also, the /dvl/products/atm directory content should contain relevant products

[examples, TBD]. WorkFlow provides a graphical representation of the level 4 processing

status.

3.4.6 Log files

ATM software has various logging capabilities. This includes notifications via e-mail, on-

screen output, logging information into a log file and Syslog.

For logging, Local::ATM::Log class plays the central role. All other classes wishing to do

logging use this class as a super class. Logging properties are configured in the ATM par

files. The default global log configuration is in

config/log/log.par,

config/log/file/file.par,

config/log/syslog/syslog.par,

config/log/email/email.par,and

config/log/screen/screen.par.

3.4.7 Maintenance procedures

At the time of writing no maintenance procedures exist. It is not known if there are any

recurrent tasks that require maintenance procedures.

IDC/DEVLAN/OG-ATM

May-08

3.5. Troubleshooting Procedures



3.5.1 Interpreting log and error messages

• The logging area for all ATM processing applications is /dvl/logs/atm. Some

applications write log files into subdirectories there, while others send status or error

messages by email to the configured ATM operator address instead of writing to log files.

cron jobs automatically remove log files after a certain delay, usually 14 days.

• Some applications use log directories, which are separate from the common log directory

described above. Most of these applications have their own log file management. These

logs can be found either in the data directory or in the software directory of the

application.

• Routine checks and monitoring:

Check if the log files expected for each application exist for the current date and also for

the previous few days. A missing daily log file usually indicates that something was

wrong with the corresponding run of the application.

Check if the partitions (logs and software) where the log files reside have sufficient free

space for the next daily log files. Under normal conditions the log files are automatically

purged, such that the total used space on these partitions does not grow. However, a full

log or software partition will typically lead to widespread logging problems in the entire

processing system.

To find the log entries for a particular processing interval, see the monitoring section of

this document. The log files are normal text files and can be read using any text editor or

viewer.

• Interpreting error messages and resolving problems:

The specific log and error messages vary for each application. Refer to the ATM

documentation for the application or subsystem to find more application-specific

information.

Error messages may indicate configuration problems, e.g. if a variable is not found or

cannot be read. In this case check the application configuration file and correct the

configuration.

Error messages may also indicate the unavailability of system resources, e.g. if the file

server is unavailable, directories or files cannot be accessed, the network connection is

unavailable, etc. Such problems may be transient, i.e. they will vanish after a short time,

or they may be persistent. Manually check the availability of the relevant system

resources. For example, if a file cannot be read check if the file exists and what

permissions are set:

ls -l <directory_path>/<filename>

All applications are run by atmops, thus this user needs to have the relevant

read/write/execute permissions. If a persistent problem with accessing system resources is

determined contact the system administrator to diagnose the specific system problem.

IDC/DEVLAN/OG-RN

May-08

Error messages may indicate the inability of an application to process specific data. In this

case the problem can be data-related (e.g. corrupt data file) or software-related. To further

determine the nature of the problem try to reproduce it for different data and determine the

specific features of the data that may trigger the problem. If the problem is clearly data-

related, the problem and the relevant data should be logged and reported for further

assessment. If the problem is related to a software bug report the software problem. If the

problem can be related to specific features, which may routinely occur in regular data

intervals, report an enhancement request.

In addition to the application log files the Syslog files on the central file store may be

helpful in specific cases. For example the cron log file can be of particular interest to

discover cron-related problems. These files are maintained by the system administrator

and are not further described in this document.

3.5.2 Solving common problems

At the time of writing no common problems relevant to the ATM software have been

observed in the Development LAN. [TBD]

3.5.3 Handling unknown problems

For individual failed processing runs, analyse the log files for the particular days to determine

the type of the problem. Try re-processing the interval if meteorological data is still available.

There is a chance that the problem was of a transient nature.

Check the scripts that were called by cron, and try to execute the relevant commands from

these on the command line as the atmops user.

The ATM scripts can also be run in debug mode with verbose output on what commands are

exactly executed. This way it may be easier to discover where the problem occurs. This is

done by executing the scripts through the shell with the –x argument, i.e. for a C Shell script:

csh –x ${scriptname}.csh

IDC/DEVLAN/OG-ATM

May-08

TERMINOLOGY

Glossary

ClearCase branch A set of sequentially numbered versions within the version tree of a ClearCase

object. All version trees have a main branch and can have additional branches

with unique branch names.

ClearCase label A label that is applied to an individual version of a ClearCase object. Various

labels are used for the Development LAN to define versions, which are part of

a software baseline or patch as well as to define versions that are currently

installed in the runtime system.

ClearCase

version tree

The set of all versions of a ClearCase object (i.e. a directory or file), which are

stored in ClearCase. The version tree starts with version 0 on the main branch

and can have additional branches each holding sequentially numbered versions.

At the IDC the ClearCase version tree holds the history of files and directories

on all IDC LANs as well as all development versions. A ClearCase version

string consisting of branch names and the version number uniquely defines

each version in the version tree.

ClearCase view A configured set of rules in ClearCase to select unique versions from all

version trees of objects in a given VOB. Individual views are identified by

view names. The views used for the Development LAN select versions based

on their branch names or on specific labels.

ClearCase VOB A logical area in ClearCase to manage a set of ClearCase objects including all

their versions. The IDC ClearCase VOBs contain high-level system directories

including all their subdirectories, files and links. Each ClearCase VOB

corresponds to a directory on the runtime system.

cron job Cron is a utility of the UNIX/Linux operating systems to automatically run

individual commands, scripts or applications at configurable regular times. All

IDC scripts and applications, which are not controlled by the DACS or by

other scripts or applications, are controlled by cron. Individual cron jobs can

be configured in the local crontab file for individual authorised users on

each machine.

data Files, which are written by the IDC processing system, are considered data if

they are re-used by the system for further processing. Otherwise they are

considered products. This categorisation is used in the Development LAN

directory structure. Data and product files are not under ClearCase version

control.

Development

LAN

The computer hardware and infrastructure used to integrate and test software

and configuration changes as well as new software at the IDC before

promoting the changes to the Testbed LAN for operational testing. Physically

separate from the Operations and Testbed LANs and under less rigorous

configuration control.

IDC/DEVLAN/OG-RN

May-08

devlan branch The devlan_view is configured to automatically create a new version on the

devlan branch if a directory or file is modified and checked-in in the

devlan_view. The latest sequential version on the devlan branch is

always the latest version that has been created under the devlan_view. If

the version tree of a ClearCase object does not have a devlan branch this

ClearCase object has never been modified and checked-in in the

devlan_view.

devlan_rn_vi

ew The devlan_rn_view is the ClearCase view to be used to modify and

check-in versions of Radionuclide software-related ClearCase objects for the

Development LAN. It selects versions based on the same branch names as the

standard devlan_view. However, the devlan_rn_view enables users to

check-out and check-in versions in the Radionuclide software ClearCase VOB.

devlan_solar

is_view The devlan_solaris_view is exclusively used to build Linux compatible

software on Solaris platforms for backwards compatibility reasons. It is

configured to check-in versions on the devlan_solaris branch. Since

source code is uniformly maintained on the devlan_branch for both Solaris

and Linux platforms the view will select the latest source code version on the

devlan branch if this branch exists in the version tree of the ClearCase

object. If no devlan branch exists for a particular objet the view will use the

same selection rules as the standard devlan_view. Only ClearCase objects

which are installed via the Solaris build procedure in this view are supposed to

have a devlan_solaris branch.

devlan_view The devlan_view is the standard ClearCase view to be used to modify and

check-in versions of ClearCase objects for the Development LAN. It is

configured to select versions based on their branch names. It will select the

latest version on the devlan branch if this branch exists in the version tree of

the ClearCase object. It will select the latest version on the R3_tst branch if

the devlan branch does not exist. If neither a devlan branch nor an

R3_tst branch exist in a given version tree the devlan_view will select

the latest version of the ClearCase object on the main branch. This latter case

occurs for files or directories that have never been modified either on the

Development LAN or on the Testbed.

DEV_LAN label The DEV_LAN label defines ClearCase versions, which are currently installed

in the Development LAN runtime system. It can only be applied to a single

unique version in the version tree of each ClearCase object. The DEV_LAN

label is manually applied when a new version is promoted from ClearCase to

the Development LAN runtime system.

DEV_LAN view The DEV_LAN view is configured to select only versions, which have the

DEV_LAN label applied. If the DEV_LAN labels are correctly set the DEV_LAN

view will show exactly the same versions as the Development LAN runtime

system directories.

Operations LAN The computer hardware and infrastructure used for operational data processing

at the IDC.

IDC/DEVLAN/OG-ATM

May-08

products Files, which are written by the IDC processing system, are considered products

if they are not further processed and are made available to (external) users.

Otherwise they are considered data. This categorisation is used in the

Development LAN directory structure. Data and product files are not under

ClearCase version control.

runtime system The directories and files, which are mounted on the machines of an IDC LAN

and which are used to operate the IDC processing software. All directories and

files in the runtime system, which are under version control, have a counter

part in ClearCase. The Development LAN runtime versions correspond to

ClearCase versions that have the DEV_LAN label applied. Data files, products

and log files are not under version control and exist only on the runtime

system.

Testbed LAN The computer hardware and infrastructure used to test new stations and

software changes in an operational environment before installation in the

Operations LAN. A close copy but physically separate and independent from

the Operations LAN.

Field of Regard Sample specific ATM Product attached to the RRR. The different FORs

(binary, quantitative, differential, integral) describe a set of predefined views

on the SRS fields. For a detailed definition see [BWD04, Annex II].

Possible Source

Region

Measurement Event (Scenario) specific ATM Product. For the PSR generation

an inversion problem is solved whereby source hypotheses are tested across the

globe and a certain period of interest. The inversion algorithm combines those

SRS fields relevant to the source hypothesis formulated and returns a time

dependent global distribution of correlation coefficients.

Source Receptor

Sensitivity Fields

(SRS fields)

Sample specific field that contains the time dependent sensitivity of the

measurement to potential sources in the measurement stations environment

during and n days prior to the collection stop time. n is currently 14 (but

planned to become 21 for full scale ATM operations). As the sample specific

sensitivity fields are stored within a standard 3-hourly-time stagger, they can

be combined to a source receptor matrix valid and daily updated for the whole

IMS RN network.

Source Receptor

Matrix

Event specific matrix consisting of those source receptor sensitivities that are

assembled for the IMS RN network according to the actual source hypotheses

formulated. This event specific matrix constitutes the RN analogue to the

"ground truth" information assembled for the location of seismic events.

IDC/DEVLAN/OG-RN

May-08

Abbreviations

ATM Atmospheric Transport Modelling

CCB Configuration Control Board

CDS Continuous Data Subsystem

CI Computer Infrastructure section

CIN Change Implementation Note

CPAN Comprehensive Perl Archive Network

CSCI Computer Software Configuration Item

CRP Change Request Proposal

CTBTO Comprehensive Nuclear-Test-Ban Treaty Organisation

DACS Distributed Application Control Subsystem

DBI Database Interface

DCR Development LAN Change Request

Dev LAN Development LAN

ECMWF European Centre for Medium-Range Weather Forecasts

EDM Ensemble Dispersion Modelling

FOR Field of Regard

FTP File Transfer Protocol

IDC International Data Centre

IMS International Monitoring System

LAN Local Area Network

LEB Late Event Bulletin

NCAR National Center for Atmospheric Research

NCEP U.S. National Centers for Environmental Prediction

NDC National Data Centre

PSR Possible Source Region

PTS Provisional Technical Secretariat

REB Reviewed Event Bulletin

RN Radio-Nuclide technology

RRR Reviewed Radionuclide Report

SEL Standard Event List

SFTP Secure File Transfer Protocol

SHI Seismic, Hydro-acoustic and Infrasound technologies

SI Software Integration unit

SRS Standardized Source-Receptor Sensitivity

IDC/DEVLAN/OG-ATM

May-08

TBD To be determined, to be done

TBV To be verified

VOB Versioned Object Base

WEB-GRAPE Web-Connected Graphics Engine

WMO World Meteorological Organization

IDC/DEVLAN/OG-RN

May-08

REFERENCES

[ATM-batchmodeATM_1.0-SIP] batchmodeATM 1.0 Software Installation Plan.

[ATM-NCEPDATA_V2.0] ATM NCEPDATA 2.0 Software User Guide.

[ATM-plot-FOR_2.0] ATM plot-FOR 2.0 Software User Guide.

[ATM-SRSM_1.0] ATM SRSM-Model 1.0 Software User Guide.

[B+07] Andreas Becker et al., Global backtracking of anthropogenic radionuclides by means of a

receptor oriented ensemble dispersion modelling system in support…, Atmospheric

Environment, 2007, doi:10.1016/j.atmosenv.2006.12.048.

[BWD04] Andreas Becker, Gerhard Wotawa and Lars-Erik De Geer, Review on New PTS modelling

capabilities supporting the emerging CTBTO-WMO response system including a proposal for

standardised model intercomparison. World Meteorological Organization, Geneva,

Switzerland, March 2004, CBS/ERA-CG/INF.1/Doc.8.3 (8.III.2004), http://www.wmo.ch/web/www/ERA/Meetings/ERACG-Geneva2004/Doc8-3.pdf.

[DSRT05] Roland Draxler, Barbara Stunder, Glenn Rolph, and Albion Taylor, HYSPLIT4 User's

Guide, Version 4.7, Air Resources Laboratory, National Oceanic & Atmospheric

Administration, 1 November 2005,

http://www.arl.noaa.gov/data/web/models/hysplit4/win95/user_guide.pdf.

[Edw03] Tryggvi Edwald, Using syslog at CTBTO, Version 0.9, International Data Centre,

Provisional Technical Secretariat, CTBTO, Vienna, Austria, 30 May 2003.

[IDC/DEVLAN/OG-RN] Development LAN Operator’s Guide -RN

[IDC/DEVLAN/OG-SHI] Development LAN Operator’s Guide - SHI

[IDC-CCB] IDC CCB Procedures, http://idc030.idc.ctbto.org:8000/Docs/CCB-Documentation.html.

[IDC-OM] IDC Operational Manual, CTBT/WGB/TL

[IDC-SCD] Software Checklist Development LAN, Version 2.1, IDC/CI, 2004

[Pre02b] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization, IDC

Software Documentation Framework, International Data Centre, Provisional Technical

Secretariat, CTBTO, Vienna, Austria, 29 March 2002.

[Pre02c] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,

CTBTO Editorial Manual, Conferences Services Section, Division of Administration,

Provisional Technical Secretariat, CTBTO, Vienna, Austria, 10 April 2002.

[Pre03] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization, New

ATM software layer 2: Software package SRSM-Model Version 1.0, International Data

Centre, Provisional Technical Secretariat, CTBTO, Vienna, Austria, 10 February 2003, http://kuredu.ops.ctbto.org/librarybox/idcdoc/downloads/SRSM-

MODEL_1_0.pdf.

[Pre04a] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization, New

ATM software layer 1: Software package NCEPDATA Version 2.0, International Data Centre,

Provisional Technical Secretariat, CTBTO, Vienna, Austria, 16 April 2004,

http://kuredu.ops.ctbto.org/librarybox/idcdoc/downloads/NCEPDATA_2_0.pdf.

[Pre05b] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,

New ATM software layer 4: Software package plot-FOR Version 2.1, International Data

Centre, Provisional Technical Secretariat, CTBTO, Vienna, Austria, 8 November 2005,

IDC/DEVLAN/OG-ATM

May-08

http://kuredu.ops.ctbto.org/librarybox/idcdoc/downloads/PLOTFOR_2_1.p

df.

[Pre07b] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,

ECMWFDATA V2.0 on demand Software User Tutorial, CTBT/ECMWFDATA V2.0/SUT,

International Data Centre, Provisional Technical Secretariat, CTBTO, Vienna, Austria, 27

January 2007,

J:\idc projects\DOCUMENTS\Radion_Devel\ATM\layer1\ECMWFDATA_2.0\SUTondeman

d.doc.

[Pre07c] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,

ECMWFDATA V2.0 ops Software User Tutorial, International Data Centre, Provisional

Technical Secretariat, CTBTO, Vienna, Austria, 27 January 2007,

J:\idc projects\DOCUMENTS\Radion_Devel\ATM\layer1\ECMWFDATA_2.0\SUT_ops.doc.

[Pre07e] Preparatory Commission for the Comprehensive Nuclear-Test-Ban Treaty Organization,

WEBGRAPE_1.1.2beta Software Installation Plan, IDC/WEBGRAPE_1.1.2_beta/SIP,

International Data Centre, Provisional Technical Secretariat, CTBTO, Vienna, Austria, 22

February 2007,

J:\idc projects\DOCUMENTS\Radion_Devel\ATM\layer4\WEB_GRAPE\WEBGRAPE_1.1.2

beta\WEBGRAPE_1.1.2beta\docs\WEBGRAPE_SIP_20070222.pdf.

[SS02] Andreas Stohl and Petra Seibert, The FLEXPART Particle Dispersion Model – Version 5.0 –

User Guide, 9 August 2002, pp. 48–64.

[W+03] Gerhard Wotawa et al., Atmospheric transport modelling in support of CTBT verification –

overview and basic concepts, Atmospheric Environment, 37 (18), pp. 2529–2537.

IDC/DEVLAN/OG-RN

May-08

4. DESIRABLE ENHANCEMENTS (TODO-LIST)

1. Add references to Web-Grape documentation.

2. Add WorkFlow snapshots and WorkFlow configuration information for the four

software layers to chapter 3.4.5.

3. Add descriptions of the cron jobs, chapter 3.3.1.

4. Add information on ATM software maintenance, chapter 3.4.7.

5. Add examples of troubleshooting scenarios, chapter 3.5.1.

6. Write section 3.4.4

7. Insert log file examples in section 3.5.1