nawips developer training

NAWIPS Developer Training

Scott Jacobs

Steve Schotz

NCEP/Central Operations

Systems Integration Branch

18-19 April 2006

18-19 April 2006 NAWIPS Developer Training 2

Overall Topics

Introductions and Schedule History / Background Development Process Coding Standards and Common Errors Library Guide and Header Files Tables and Help, PDF and Error Files GEMPAK Fundamentals Test Programs and Other Common Debug Tools Utilities and Access to SIB Systems Upcoming and Long-term projects


Schedule

4 sessionsDay 1

9:00-12:001:30-4:30

Day 28:30-11:301:00-3:30

1 or 2 breaks during each session

Ask questions!


Day 1 - Morning

Introductions History / Background Development Principles Development Process


Introductions

Name Where do you work? What do you do? Why are you here?


Topics

N-AWIPS mission N-AWIPS history N-AWIPS customers Overview of N-AWIPS functionality N-AWIPS development principles Overview of N-AWIPS development

process


N-AWIPS Mission

Develop robust meteorological workstation software to meet NCEP requirements, NCEP-AWIPS (NAWIPS)

Provide access to high resolution/global datasets

Provide flexible meteorological analysis and display tools

Provide integrated product generation capabilities for current and future modernized products

Provide robust, centrally supported and maintainable software


N-AWIPS History

GEMPAK- Developed at NASA during 1980s to support research and academic communities (foundation for N-AWIPS) Provides set of ~ 150 libraries, (reusable code) for decoding, data basing,

performing calculations, displaying, and product formatting Extensive use in university and research communities

Exchange of ideas, algorithms and codes GEMPAK migrated to NMC- Early 1990s

National Centers (NC) given responsibility to address NC specific requirements – Late 1980s Recognition of significant mission differences between WFOs and NCs Decision primarily cost driven


N-AWIPS History

Transition Project (TP) formed in Office of Director - 1991 TP mission - address NCs workstation modernization

requirements AWIPS contract awarded – 1992 TP begins deployment of NTRANS into operations – 1993 TP folded into NCO – 1995 AWIPS operational – 1999 New AWIPS contract awarded to Raytheon – 2005


N-AWIPS Customers

NCEP service centers Climate Prediction Center; Hydrometeorological Prediction Center; Ocean Prediction Center (Camp Springs, Maryland)Aviation Weather Center (Kansas City, Missouri)Storm Prediction Center (Norman, Oklahoma)Tropical Prediction Center (Miami, Florida)

NCEP supercomputer and Environmental Modeling Center

NWS Alaska & Pacific regions

NWS River forecast centers

Unidata (~200 universities + private industry)

Government labs (MDL, ARL, ...)


High Level Requirements

Diverse products, geographic and temporal scales

Somewhat diverse forecast processes

Mostly common/generic software requirements


N-AWIPS Fundamental Capabilities

Data visualization

Integrated product generation


N-AWIPS Data Visualization Capabilities

Definition: Ingest, decoding, calculation and display of

meteorological data

Meteorological data types include: Radar and satellite imagery Model data: grids, MOS Observations: METAR, marine, aircraft, upper-air,

lightning, pilot reports, profilers, … Forecast products: flash flood, Sigmets, Airmets,

watches, warnings, ….


N-AWIPS Data Visualization Capabilities

Critical requirements include: Flexible user-defined geographic and temporal scales and map projections Wide variety of user-specified, “on the fly” calculations

Grid Diagnostics ~170 built in grid operators that allow user specified combinations of calculations, e.g., adv((vor(geo),thrm(hght@300:700))

Variety of display options, e.g., station plotting models, contour attributes, etc. User-defined data overlay options

Key display functions: Multiple animation loops: max of 16 loops with total of 120 frames Automatic data time matching to frame time to support data overlay

comparisons Procedures for automatic loading of data into multiple loops Continuous roaming to allow detailed views of data over large geographic

regions


Integrated Product Generation

Definition: Creation/Editing of graphical products overlaid on meteorological data

displays

Critical requirements: Drawing and editing tools using meteorological objects, e.g., fronts, weather

symbols, etc. Graphical objects navigated to account for multiple product sectors and

projections, wind rotation, … Facility to import first guess fields, e.g., model fields and other centers’

graphical products Product formatting to support GIF, TIFF, PostScript, GRIB, BUFR,

Redbook… Product layering to support multi-component or multi-time concurrent

editing Object grouping


N-AWIPS Development Principles

Maximize Code Maintainability Software must be written to be understood/maintained/enhanced by someone

other than original developer(s) Must be operationally robust

Maximize Code Reuse Employ functional libraries with test programs Never write code more than once to perform a particular function

Maximize Functional Flexibility/Adaptability Generic functionality to meet broad spectrum of current and anticipated needs,

e.g., flexible calculation tools, display options Maximize Hardware Independence - Unix (IRIX, SUN, HPUX, AIX), Linux, single

or multi-headed workstations Multiple hardware platform support improves software robustness Enables user platform/configuration choices Employ industry standards, e.g., ANSI C, no extensions


N-AWIPS Development Principles

Minimize dependence on commercial external software systems Keep the system as simple as possibleFacilitate installation and testingReduce user costs of N-AWIPS


N-AWIPS Development Process Overview

Use an evolutionary development strategy Build in small frequent increments, quarterly release

cycle Refine requirements based upon forecaster feedback Continue to improve system to support new products,

refinements of existing products and increase forecaster productivity


N-AWIPS Development Process Overview

Employ disciplined development process covering entire software development life cycle Repeatable process Process must be efficient to allow changes to the

system to be cost effective Traceable process and document trail with peer

reviewed check points


N-AWIPS Development Process Road Map

Document Creator Reviewer(s)

Requirements Centers Centers and N-AWIPS Team Lead (NTL), N-AWIPS Leads

Release Task List NTL N-AWIPS Leads

Task Proposal N-AWIPS Lead NTL, Lead SW Engineer (LSE), [Center]

Task Design(s) N-AWIPS Lead Lead SW Engineer (LSE) [Leads+NTL]

Config Items (Code, tables, help etc)

Coder N-AWIPS Lead(s), LSE

Delivery Form Coder N-AWIPS Lead, Configuration Manager (CM)

Code check & logs (nawips.log, changes.log, tables.log)

CM

Delivery test report Independent tester N-AWIPS Lead, NTL

Release Note Entry N-AWIPS Lead NTL

Integration Test report N-AWIPS Team N-AWIPS Lead, NTL, LSE


Development Process Topics

Requirements Analysis Proposal Detailed Design Coding and Developer Testing Independent Testing Release Notes Approvals


Requirements Analysis

Attempt to understand the user needs Must ask a lot of questions to make sure the needs are

well understood Don’t assume anything

Think about what the users are saying Ask the users to explain anything that is not clear

Don’t skip this step It always takes longer to fix something than it would

have to learn the requirements first Even for small tasks, spend some time on requirements


Proposals and Designs

Purpose is several foldReduce costly downstream errorsCreates a repeatable processDocuments the approach and design for future

referenceTies the requirement to the coding and testing


Proposals

Task Goal1-2 sentence statement of what you want to

accomplish Identify the requirement number addressed

BackgroundExplain user requirementsReview current functionality

The length of the proposal is dependent on the complexity of the task


Proposals

Proposed Solution or OptionsDescribe any algorithms If there is more than one solution, list them for

review and discussionAlso include any pros/cons to each option

Areas of Code Affected Identify the libraries or applications

E.g., All GD programs; Only GDPLOT2; DG library; PGEN


Proposals

Specific Tasks If this will be a multi-part task, list the subtasks For example

Part 1: Create new library routines and add to test programPart 2: Incorporate new routines in applications

Each subtask should be able to be tested alone Test Plan

Create a preliminary test plan Will help you make sure that you have covered

everything This will also help the reviewer(s)


Proposals

Proposals are reviewed by Lead Software Engineer (LSE) NAWIPS Team Lead (NTL) Other designers, as needed In some cases, users or their representatives

Discussion If needed, a meeting will be arranged to discuss various aspects of

the proposal Approval

A proposal is approved when either the LSE or the NTL initials the document

Approval may be contingent upon changes indicated in the body of the document

See example document


Detailed Design

DescriptionOverview of the changesSummarize the proposalLength is dependent on the complexity of the

taskDesigns are also done for bug fixes

In this case reference the bug or problem report


Detailed Design

Function List Include any calling sequences changes (CSC)

Include the entire sequence with the changes If no CSC, only indicate “No CSC”

Describe the changes needed for each function Include enough detail so that someone other than you

could do the coding Also include any functions to be deleted


Detailed Design

Library Check This is important when changing GEMLIB/CGEMLIB functions Many applications call these library functions Changing a library routine for one task may have unintended

consequences in some other application This check attempts to find all locations where a particular

function is called in order to assess the impact This is also needed to ensure that all necessary testing is included

in the test plan The utility “whocalls” can be used to help fill this in

Utilities will be discussed more fully later


Detailed Design

Header Files, Tables, etc. List any files, other than source code, that also need to

be modified or created Test Plan

Include all tests that must be run Test the intended changes Test any applications found from the Library Check to

ensure no change in functionality


Detailed Design

Author Make sure to put your name in as the author

Update the Level of Effort This should be a number of days to finish the coding,

unit testing and independent testing Update the start and end dates

These dates are for when you start and finish editing the detailed design

If the design is returned without approval, and you modify the document, indicate that as “Version 2” with the new dates


Detailed Design

Designs are reviewed by Lead Software Engineer (LSE) Others as needed

NAWIPS Team Lead (NTL) Other designers

Discussion If needed, a meeting will be arranged to discuss various aspects of

the design Approval

A design is approved when the LSE initials the document Approval may be contingent upon changes indicated in the body

of the document



Coding and Developer Testing

Copy existing files or create new files in your local account Do not work in the software account Make sure you get all necessary tables, etc. Also get the link scripts

Compile scripts are all in $GEMPAK/buildLink scripts are in the directories with the application source

code

Be careful with file names if working on GPLT functions since the same name is used in APPL and GPLT; and GPLTDEV and DEVICE



Proceed with the development as outlined in the detailed design If you find that the design is incomplete or incorrect,

update the design and make a note in the document header If you are not the original author of the design, discuss it with

that person first Make sure that you update the log in each source file

Compile on as many platforms as you can There should be no compiler errors nor warnings when

you are ready to deliver Access to SIB systems will be covered later



Conduct the developer unit tests Test the changes as outlined in the test plan Collect any special data sets for later testing Test unusual situations

That is try to break the code with strange inputThis is easiest if the new routines have been added to the

appropriate test program



Delivery Create a directory tree for your changes that mimic the

software accountStart at the “gempak” directoryPut all files in the appropriate subdirectories

Enter your name and the dates you started and finished coding on the form

Enter the path to the delivery tree you createdAccess to SIB systems will be discussed later

Describe the changes in a few sentencesThis description will become the log entry in “nawips.log”



Delivery (cont.) List all modified files

Put those with calling sequence changes first

List all new files List all files to be deleted All files listed will be compiled by the configuration

manager (CM) If a header file is changed, make sure to indicate all source

files with dependencies on that header fileThese files will also be compiled by the CM

List all applications that must be linked



Delivery (cont.) Describe the tests conducted and any significant results

Indicate any platform specific information

Provide a path to a set of executables for all platformsThis is for the code reviewer

Update the testing section of the Design with the followingDescribe any special data sets and their locationProvide any special testing instructions not covered in the

original design test plan



Code is reviewed by either The task owner

This is the person who wrote the design, if someone else did the coding

Another lead designer Lead Software Engineer (LSE)

Approval A delivery is approved when the reviewer initials the

document It is then given to the CM to add to the baseline

software accountThe CM enters the delivery number on the form



Independent Testing

Conducted by someone not already associated with the task Must test on all platforms List all tests done

Must do all tests in the test planAnything else the person can think of to break it

List any problems encounteredThese would indicate a serious condition and would need to be

addressed as soon as possible

Also list any observations and suggestionsThese are less serious, and may be addressed later


Independent Testing

The task owner reviews the test report Writes responses to all problems

If there are serious problems, the task goes back to coding

Writes responses to all observations and suggestions If any warrant immediate fixes, the task goes back to coding

If there are no issues or delayed issues, the task is DONE!

Approval The task owner initials and dates the test report after

review



Release Testing

Test scenarios to emulate operations List of commands and button clicks Product creation tests Modified as needed to accommodate different data sets

Batch scripts NTRANS metafile generation New scripts to test all programs

Allow comparison between versions

Can and should be run at various times throughout the release cycle


Release Notes When are Release Notes required

The user interface changesResults or output changePublic calling sequences changeAt the discretion of the NTL

Should be written as early as possible in the development process



Release Notes Should sufficiently describe the changes to

functionalityRanges from a few sentences to a few

paragraphs

Will be incorporated into the full Release Notes document and the “What’s New” help file in NMAP2

The NTL reviews and edits the release notes for consistency with the design


Day 1 - Afternoon

Coding Standards Common Errors Library Guide


Coding Standards

Principles Make the system extensible so that additional needs can

be metDon’t need to serve all needs at once

Isolate complexity as much as possible Code must be able to be maintained by someone other

than the original developerCode should be readable and consistentFunctions should be as short as possible

Utilize functional libraries to maximize code reuse Routines/functions should be independently testable

Use of test programs will be discussed later


Coding Standards

Fortran [77]All files should consist of only one subroutine

or functionLibrary routines should be well defined with a

single purpose

Lines should be less than 72 charactersDo not use the GOTO statement


Coding Standards

Fortran [77] Routine Names

GEMLIB routines are 2 character library name + underscore + 4 characters to describe routine

Program routines are 2 character program class + 1 character program id + 3 characters to describe routine

GPLT routines all start with “G” and have up to 6 charactersDEVICE routines all start with “D” and have up to 6

charactersDriver routines all start with “H” and have up to 6 characters


Coding Standards

Fortran [77] Uppercase

All Fortran keywordsSubroutine and function names Parameters defined by the PARAMETER statementThis is for readability since Fortran is case independent

VariablesLower caseShould not exceed 6 characters Implicit typing

I-N are Integers, only All others are Reals All Characters must be declared


Coding Standards

Fortran [77] Indentations

First should be a full tab (8 spaces)Second should be 4 spaces after the tab (12 spaces)Third should be 2 tabs (16 spaces)Etc.

Continuation lines should begin with 5 spaces and a plus sign (+) or ampersand (&)

All mathematical symbols should have one leading space and one trailing space, when feasibleA set of assignments should have their equal signs lined up


Coding Standards

Fortran [77] Common Blocks

In GEMPAK, there are specific uses for commonsCommons are not used to just bypass calling sequencesCommons are stored in include filesCommons group related variablesUse is discouraged for main programsMay be used for libraries – hidden from the applications

Parameter definitionsGeneral use parameters are defined in GEMPRM.PRMLocal parameters may be defined in the routine or local

include file Include files will be discussed further later


Coding Standards

Fortran [77] Comments

Should be aligned with the codeLines with text should start with “C*”Should have a blank comment before and after any textShould be as descriptive as possible, while remaining brief

See code examples


Coding Standards

C Generally the same standards as for Fortran Indentation starts in the first column

Each indent is 4 spaces thereafter

There are no continuation characters Comments are indented with the corresponding code

Open the comment with “/*” on one lineText on subsequent lines beginning with a “*”End the comment with “*/” only, on one lineThe asterisks are lined up


Coding Standards

C Groups of related functions may be collected into a

single file, called a module Every function must have a prototype

This will be discussed further with header filesFortran functions called by C must also have prototypes

C is case sensitive, case is used to distinguish contentMacros in #define statements are all uppercaseLocal variables are lower case or mixed with the first character

as lower caseLocal-global variables start with underscore and may have

mixed caseGlobal variables, from header files, do not currently follow a

standard for case, most are all lower case


Coding Standards

C All control statements (while, for, if, do) should have

the executable statements enclosed by “{“ and “}”This applies even if there is only 1 statementMakes it easier to add functionality later

Including adding diagnostic print statements for debugging

Makes it easier to understand exactly what is included in the control statement

The { should appear on the same line as the control The } should be indented the same as the corresponding

controlSee code examples


Coding Standards

General Each file should begin with a prolog to document the

routine The chronological log in the prolog should be updated

every time the routine is modified If a routine is running out of indent space, then the

routine is trying to do too muchConsider breaking it up into smaller functions

In control and math statements, do not assume order of operation, use parentheses to be explicit


Common Errors

Case sensitivitySince C is case sensitive and Fortran is not

some errors can ariseGEMPAK uses case in Fortran for readability

Even so, everything is really lower case after compilation

In C Fortran functions must be lower case Any C function called by Fortran must be only lower case


Common Errors

Value AssignmentsDon’t forget to set the value of variables before

using the valueIn C, it is preferable to do the assignment in a

separate statement from the declaration In this way the assignments can be grouped together


Common Errors

Check all cases for a variableDon’t forget 0, nor the outer limitsWhen checking values in a control statement

Make sure that all possible values are accounted for in the code

A common error is to check for positive or negative, but forget about what happens when the value is zero

Is there an upper/lower limit that makes sense for the value

Does a line width of 1,000,000,000 make sense, and can it happen in the code?


Common Errors

Underscore at the end of function name By default, without the user’s knowledge, some Fortran

compilers put an underscore at the end of function names IRIX, SunOS, Linux In fact, Linux puts 2 underscores by default

A compiler flag set for GEMPAK limits it to 1

Any Fortran function called by C, or C function called by Fortran, must have an underscore added to the function name in the C source

This is accomplished with a header file (uscore.h) and a compiler flag (-DUNDERSCORE)


Common Errors

Return codes and Error processing If a function needs to return after a problem, the return

code should be set a unique valueAll libraries and programs have error tablesWhen adding a return code, check this table for the next

available value

Errors return a negative valueErrors should cause the processing to stop and return to a

reasonable point in the application

Warnings return a positive valueProcessing can continue, but it may not be what the user really

wanted


Common Errors

Exceeding array sizesMake sure that all array references are within

the bounds of the arrayCan the value of the index ever be negative?

In Fortran, this may be ok depending on the array declaration

Can the value of the index ever be larger than the allocated size of the array?


Common Errors

Use of the qualifier “const” in CEnsures data protection

Important when passing strings or pointers that should not be changed in the function

Constant parameters in a function will generate a compiler error if there is an attempt to modify the value


Common Errors

Memory managementWhen using dynamic memory allocation in C

Make sure to free all local memory before returningIf the allocated memory is global to a library

Make sure that a function is provided to free all global memory


Common Errors

C vs. Fortran Variable types

C has variable types that are not supported by Fortran “unsigned” has no meaning in Fortran int and long may be different sizes in C

There are also types that must be declared differently by Fortran

short == INTEGER*2

C array counting starts at 0, Fortran starts at 1 Two dimensional arrays are reversed

vararr[10][20] == vararr(20,10)


Common Errors

C vs. Fortran Fortran passes all variables by reference

The equivalent in C is a pointer to a variable typeConstant values should not be used in calling sequences

StringsArrays of strings are different in C and FortranCannot pass an array of strings between the two languagesFortran automatically passes the length of all strings in a call,

hidden at the end of the list When calling Fortran from C with a string in the call, add a

“strlen(strng)” for each string after the last real sequence parameter


Common Errors

C vs. FortranPointers

If you are used to Fortran the concept of pointers can be difficult to follow

In C, you can access both the value and the memory address of a variable

A pointer “points” to the memory locationExample

int *ivar;*ivar is the value (dereferenced from the pointer)ivar is the memory address


Library Guide

GEMPAK External


GEMPAK Libraries

The source is organized by function General library functions (C and Fortran) Decoder libraries Message queue utilities GPLT Device drivers Specific program code


GEMPAK Libraries

Some libraries have their own global variables not shared with other libraries

Fortran does not allow for private functions Therefore, it is by convention that some functions are not called

from other functions outside of a particular library

In C, defining a function with the “static” qualifier will make it local to a module Therefore, it cannot be called from any functions outside of the

module


GEMPAK Libraries

build data

error help

include maps

tables pdf

parm txt

utilities

appl bridge

cgemlib contrib

device driver

gemlib gplt

nmaplib nxmlib

programs syslib

textlib

source

gempak bin doc

icons os

resource extlibs

nawips


GEMPAK Libraries General Libraries (C and Fortran)

The GEMPAK libraries handle a number of functions that are needed by all programs

String manipulations File access Table access Time calculations Data packing and data file management

There are also libraries that are used by certain classes of programs Grid diagnostics Surface data access Upper air data access Image data access and display GRIB processing


GEMPAK Libraries

aw cv

dg, de, df, dl, dv, pd dm

dp er

fl gb

gd gdu

gg gh

gp gr

im in

ip jb

lc lv

lw mr

mv na

oa pc, pr, ps, pt

sf sn

ss st

tb tg

ti tm

vf

gemlib


GEMPAK LibrariesLibrary Name

Function Usage

AW Display Redbook graphics files Used by GPMAP

CV Curve functions

DG, DE, DF, DL, DV, PD

Grid diagnostics Public functions called by applications; Private functions called internally to compute diagnostics

DM GEMPAK data management Called by SF, SN and GD

DP Data packing Called by DM

ER Error message processing and display

FL Low-level Fortran file access

GB GRIB data processing Used by NAGRIB

GD Grid data interpretation Public functions to access grid data

GDU Grid data utilities for matching wild cards Used by GDINFO

GG Graphical display functions for maps, lat/lon, misc data

GH Hurricane product creation

GP Generic functions pulled from GPLT Used to transform wind directions



Function Usage

GR Grid data utility functions Functions that don’t fit other groups

IM Image data processing

IN Input string parsing Called by applications

IP GEMPAK text interface Get the user input for all applications; NTS files

JB Jack Woolen BUFR functions Previous version of NCEPBUFR library

LC Location functions Processes AREA and GAREA parameters

LV Level utilities Used in the upper air applications

LW Reads a “long-word” format file Only lightning data is stored in this type of file

MR Upper air merge functions Used in the upper air applications

MV Byte swapping and manipulation When the file was saved on a different machine

NA NAGRIB library

OA Objective analysis

PC, PR, PS, PT

Parameter conversions Public functions to set up later computations; Controlled by table: pcconv.tbl



Function Usage

SF Surface data interpretation Public functions to access surface data

SN Upper air data interpretation Public functions to access upper air data

SS System functions Before C libraries, allowed access to system level functionality

ST String manipulation

TB Table reading Various specific table access functions

TG Grid time calculations

TI Time calculations

TM Terminal input General functions to get input from the terminal

VF Create SPC watch products Works on the “verification” file to create products; Written in C, but part of GEMLIB


GEMPAK Libraries

cas cbf

cds ces

cfl cgr

clo cpg

crg css

cst ctb

cvg cvq

g2 shp

spf utf

xml

cgemlib



Function Usage

CAS Reads intermediate ASCII files for SIG WX charts VG to BUFR and BUFR to VG processing

CBF Accesses BUFR files for SIG WX charts

CDS Display of VG objects PGEN and display in GPMAP

CES VG object attribute management PGEN

CFL Low-level C file access

CGR Miscellaneous graphical utilities Many for PGEN and other polygon manipulation

CLO Locator functions Location searching in station or bound files

CPG Plot 6-bit FAX files Used by GPFAX

CRG Range record management PGEN

CSS System functions Similar to Fortran SS

CST String manipulation

CTB Table reading



Function Usage

CVG VG file access

CVQ Query information about VG files and objects

G2 GRIB2 data processing Used by NAGRIB2

SHP Shape file processing Used to convert Shape files to GEMPAK map and bounds files

SPF Stored Procedure File processing Used by NMAP2 for saving and restoring lists of data sources

UTF AFOS graphics display Used by GPMAP

XML Interface functions to XML libraries


GEMPAK Libraries

Decoder libraries Each decoder has a library of functions that is unique to that

decoder METAR Synoptic Upper Air MOS Etc.

There is also a library of routines that are used by all decoders to handle common functionality, e.g., command line input, bulletin reading


GEMPAK Libraries

Message Queue Library Communication between GEMPAK processes is done using

message queues There is a library of routines that is linked with the application

programs that construct the information for the message queue This APPL information is sent to the GPLT process Then GPLT has a library of routines that creates messages for a

second message queue The GPLT information is sent to the user specified device driver


GEMPAK Libraries

GPLT Mainly responsible for coordinate transformation from Map

coordinates to the specific Device coordinates Grid – Map – Plot – View – Normal – Device

Other responsibilities Setting and storing the current map navigation Satellite navigation computations Contour and streamline algorithms Wind rotation Setting plotting margins Setting and storing object plotting attributes

For example -- Lines: Dash pattern and width


GEMPAK Libraries

access attribute

bounds color

contour control

coord curve

device graph

grid main

map plot

satellite transform

utility

gplt


GEMPAK LibrariesLibrary Name Function

ACCESS Start and close GPLT

ATTRIBUTE Set and query attributes of drawables (e.g., lines, fronts)

BOUNDS Set and query coordinate boundaries

COLOR Set and query color information

CONTOUR Compute and display contour lines and fills

CONTROL Control frame information (e.g., clear, roam, looping, windows)

COORD Private functions to compute transformation information

CURVE Sets and computes the type of contour curves (e.g., cubic splines)



GRAPH Set and query information for plotting graphs

GRID Set and query information for setting grid projections on maps and graphs

MAIN The GPLT main program

MAP Set and query information for plotting maps

PLOT Plot the drawables

SATELLITE Native satellite navigation transformations

TRANSFORM Coordinate transformation functions

UTILITY Draw maps, lat/lon lines, streamlines and polygon boundaries


GEMPAK Libraries

access attribute

color control

curve draw

main plot

satellite window

device



ACCESS Start and close GPLT

ATTRIBUTE Set and query attributes of drawables (e.g., lines, fronts)

COLOR Set and query color information

CONTROL Control frame information (e.g., clear, roam, looping, windows)

CURVE Sets and computes the type of contour curves (e.g., cubic splines)

DRAW Device independent code to interface to the device drivers

MAIN The DEVICE main program

PLOT Plot the drawables

SATELLITE Plot images to the drivers

WINDOW Set the “window” dimensions and the clipping boundaries


GEMPAK Libraries

Device Drivers Current device drivers

FAX NCEP 6-bit Fax GF GIF file created using the X server GIF GIF file created without using the X server NC NCEP Computer Graphics Metafile PS PostScript RBK AWIPS Redbook graphics file TIFF TIFF graphics file VG Vector Graphics file XW X window


GEMPAK Libraries

Application Programs Decoders Grid data plotting Surface data plotting Upper Air data plotting Objective Analysis Graphic utilities Graphical User Interface programs

NMAP2 combines display of image, grid, surface, upper air, misc


External Libraries

Collection of libraries from disparate sources Collaborative efforts

Automated Objective Dvorak Technique (AODT)GAMET

Specific functionalityGeneric Polygon Clipper (GPC)MELBUFR / NCEPBUFRCREWSSNetCDF

General utilitiesXML / XSLT / DOMPNGJASPER


External Libraries

The decision was made to pull these libraries out of the core GEMPAK directory treeMade more sense to keep them separate from

locally developed code

The installation of some of these libraries forced the recent change in the location of executable, library and include files to the “os” subdirectories


Day 2 - Morning

Header Files Tables Maps and Bound Files Help, PDF and Error Files Build Scripts GEMPAK Fundamentals Test Programs


Header and Include Files

Global C header files and Fortran Include files for GEMPAK are in $GEMINC

System specific files and all header files from the external libraries are in $OS_INC

Library specific files are found in the directory with the source code

See file listing



Fortran Include files Common block files

DGCMN.CMNsfcmn.cmn

Parameter filesSets constant values used to declare arrays or string lengthsGEMPRM.PRMFUNCCODE.PRM

Files in $GEMINC are those needed by functional groups of code that are not in the same source directoryDGCMN is used by DG, DF, DV, DE, DL which are all

related to grid diagnostics



C header files Macro definitions Structure definitions Global variable declarations Public prototypes Files in $GEMINC are those needed by functional

groups of code that are not in the same source directorygeminc.h gemprm.h, proto.h, uscore.h, vgstruct.h

Always use the “file.h” construct for local files, not <file.h>This allows a local copy to override the global copy, during

development


Tables

Tables are all in $GEMTBL Directories are an attempt at categorizing

the tablesOrganized by function or type

bufr,colors, config, draw, grid, loop, luts, pack, parms, pgen, rad, sat, stns, txtprd, xsd, xslt

Organized by program or librarygrphgd, hcnadv, melbufr, nmap, nsharp, nwx

See file listing


Tables

There is a hierarchy used when searching for a table ./table_name (local directory) $NCDESK/type/table_name $NCSITE/type/table_name $GEMTBL/type/table_name

NCSITE and NCDESK are used to hold site-wide or user specific tables

Copy the tables from $GEMTBL to these locations before modifying

Having the local directory first facilitates testing since it will override the other locations


Tables

Key tables In the config directory

prefs.tbl Settings that are used to control default values Most of the tags refer to the user interface or product generation

in NMAP Can be used for any program

datatype.tbl Allows the use of aliases for data sources Equates the aliases to paths and file name templates Sets other attributes for the data source

clo.tbl Sets up aliases for the bounds and stations tables used in the

locator library


Tables

Key tables In the config directory

forecasters.tbl List of forecasters to display in the user interface for selection

mapfil.tbl List of map aliases

miscset.tbl Sets the attributes and flags for the miscellaneous data sources Sets default values

printer.tbl List of printers for selection in the user interface

prmlst.tbl Parameter list aliases Sets the list of parameters to display in a station model Sets the colors and other attributes


Maps

The map files can be found in $GEMMAPS GEMPAK uses a internally developed direct access file

format $GEMMAPS/ascii/mapfiles.doc gives specifics

The majority of the maps come from the NWS Shape file database and are converted to GEMPAK map files

Some maps came from the CIA map database Some maps have been created from VG files Most maps have multiple resolutions

TP, HI, ME, LO


Maps

The maps can be displayed in any GEMPAK plotting application or NMAP2

To allow NMAP2 to access the maps Edit the file $GEMTBL/config/mapfil.tbl to add an

alias for the particular map Edit the file $GEMTBL/nmap/mapovl.nmap to add the

alias to the Map control window The GEMPAK programs can use the alias or the

actual file name Multiple maps can be overlaid by listing them in the

parameter $MAPFIL separated by a “+”


Maps

The map utility programs are in $GEMMAPS/programs There are programs to convert from the binary format

to ascii and vice versagsfssf and ssfgsf If a map needs to be fixed: convert it to ascii, fix the problem,

then convert back to binary

The “nam” programs work with the CIA map data “reducebnd” is used when creating boundary tables


Bounds Files

These files are located in $GEMTBL/bounds They are not really tables, but…

The files are plain ascii files There is an index file that is created by the utility

“createbinfo” Any time that the bound file is modified, the index file

must be updated Bounds files are closed polygons that represent

geographical regions or forecast areas A few of the NWS Shape files are also converted to

bounds files Some files are created from VG files using “vgftobnd”


Bounds Files

The bounds files are used in numerous functionsThe NMAP2 locator (bottom right of the

window)GFA processing for the AWCWatch product creation for the SPCUnified Surface Analysis VG file clipping Ice extent display for the OPCDisplay of filled areas from GPMAP using the

BND parameter


Help, PDF and Error Files

All help files are in $GEMHLPThe program files are in the “hlp” subdirectoryThe parameter file are in “hlx”

Users can display the content of the help files from the various programsFrom a GEMPAK program

Enter any program or parameter name after the word “help”

All GUI programs have a “Help” button that will display the file(s) related to that application



Parameters require 2 files .hl2 files contain the content displayed to the

user .hl1 files are a one-line description used in the

program help files and the PDF files Program help files are generated from the

TXT filesThe text files are in $GEMPAK/txt/programsThe help files are generated by the utility

program “hlpbld”



PDF files are unique to GEMPAKNOT the Portable Document Format

Cannot use Acrobat to display the contentsJust an ascii file

Left over from the time that GEMPAK used the TAE text interface developed at NASA

Format was changed from the original TAE to just a collection of the all of the .hl1 files for a program

Created using the utility “pdfbld”



Error files Found in $GEMERR One for every GEMPAK application and library Contain a list of code numbers and descriptions

Negative values are errorsPositive values are warnings

Errors are displayed to the user using the function ER_WMSGThe function uses the category, the code number and an

optional string to locate and display the messageThe optional string replaces “!AS” in the description in the

error file


Build Scripts

Compile scriptsLocated in $GEMPAK/buildUsed to compile librariesAlso compiles application librariesAdds all compiled files to the appropriate

library archive (.a) file


Build Scripts

Link scriptsUsed to link all applicationsDriver scripts are in $GEMPAK/build

These scripts go to the individual program directories to link the applications

The link scripts are located in the program directories


Build Scripts

Things to considerAs the name implies, nmaplib should only be

used when linking NMAP2Many link scripts have GEMLIB or CGEMLIB

listed more than onceThere is no way to predict when or how to add more

than one reference to a libraryWhen in doubt, follow the link script for a similar

programStill no luck, send e-mail or ask


Dot Files

Used to set environment variables useful in the execution of the applications

Also set system aliases and the exec path Many environment variables, and the path, are set

based on the current operating system In csh, the user sources the .cshrc In sh, bash or ksh, the user sources the .profile

Changes are rare, but very important to keep up to date Modifications are highlighted in the release notes


GEMPAK Fundamentals

Functional Layers Applications GPLT Device Drivers Decoders GEMPAK Data Files Product Generation and VG files


Functional Layers of GEMPAK

Application AP

PL

Message Queue

Message Queue

Normal execution of an application, that plots data, starts GPLT and a Device Driver

The processes communicate using Message Queues SYSLIB is used to create and use the Message Queues

Allows user to change Device Driver during execution of a program Allows user to overlay the output of many apps to 1 Device Driver

GP

LT

GP

LT

DE

V

DE

VIC

E

DR

IVE

R S

peci

fic

H-l

evel

DR

IVE

R


Example: Drawing a Line

Application AP

PL

Message Queue

Message Queue

Same function names in APPL/GPLT and GPLTDEV/DEVICE APPL sends message through MQ GPLT main program receives message and calls function Similar from GPLT to DEVICE

GP

LT

GP

LT

DE

V

DE

VIC

E

DR

IVE

R S

peci

fic

H-l

evel

DR

IVE

R

GSLINE

GLINE

GSLINE DSLINE DSLINE

GLINE DLINE DLINE

ILINE

HLINE

XLINE


Example: Without MQs

Application

Executable file is larger Can only output to one device driver Done for better performance in batch processing Done for GUI programs, since the input mechanism is the same as the

output device

GP

LT

DE

VIC

E

DR

IVE

R S

peci

fic

H-l

evel

DR

IVE

R

GSLINE DSLINE

GLINE DLINE

ILINE

HLINE

XLINE


GEMPAK Applications

Most applications are divided based on the type of GEMPAK data file they work with GD – Grid data processing SF – Surface data processing SN – Sounding (Upper Air) data processing

Others are grouped by functionality OA – Objective analysis GP – Miscellaneous graphics NA – Binary data decoding (GRIB) DC – Decoders GUI – All graphical user interface programs


GPLT

Responsible for the following functionalitySetting MAP and GRID projectionsCoordinate transformationsContouring and Streamline algorithmsNative satellite navigation computationsSetting marginsSetting and saving plotting attributesFront-end to plot/draw functions


GPLT

Coordinate Systems Grid

IntermediateQ – rotated

MapW – rotatedLinear

Plot View Normal Device

Units Grid is grid point

based on kx and ky Map is lat/lon in

degrees P, V, N are

normalized and range from 0 to 1

Device is pixels in the output – i and j


Device Drivers

Device independent routines plot the following Symbols and markers Winds as barbs or arrows Text Pattern lines Filled polygons Fronts

All of these types of elements are converted to line segments for output to the specific driver

Clipping to the plot boundaries is also done in the device independent code


Device Drivers 3 common blocks are used to set and store

drawing attributes DEVREQ

Attributes requested by the userUsed by GPLT

DEVSETAttributes used in the deviceUsed by both GPLT and DEVICE

DEVACTAttributes actually set in the deviceUsed by both GPLT and DEVICE

For example: There are 32 colors. If the user requests color 40, the actual color used is 8.


Current Active Device Drivers FAX – NCEP 6-bit Fax format GF – GIF image (extension of XW driver) GIF – GIF image (no X server needed) GN – Generic test driver NC – NCEP Computer Graphics Metafile PS – PostScript RBK – Redbook AWIPS graphics TIFF – Tagged Image File Format VG – Vector Graphics File XW – X Window XWP – X Window Plus (combines XW and PS)


Device Driver Differences

FAX, RBK and TIFF are monochrome only PS can be monochrome, gray shades or full color The origin for most drivers is the bottom left

corner The XW/XWP origin is the top left corner

VG does not have a DEVICE coordinate All locations are saved to the file as lat/lon values

VG also does not draw the map to the output file


Decoders

In general, a decoder is a program which reads data in one format and converts it to another formatText report to GEMPAK data file

GEMPAK decoders also take care of opening output files and writing data to the files


Decoders

The decoders take command line options and input

Help is displayed with the “-h” option The output file name may use a template for

the date and timeThe actual file name will change depending on

the input data


Decoder OptionsOption Description

-v N Level of verbosity for the logs

-c curtim Set the “current” time

-b nhours Number of hours to decode prior to “current” time

-d decoder_log Decoder log file name

-t time_out Interval for the timeout in seconds

-n Flag to NOT save the text report as data

-h Print the help file, then exit

-p prmfil Set the parameter packing table

-s stntbl Set the station table

-a iadstn Set the number of additional stations

-m maxtim Set the maximum number of times


Current Decoders

DCACFT – Aircraft DCAIRM – AWC AIRMET DCCSIG – Convective SIGMET DCFFA – Flash flood watch DCFFG – Flash flood guidance DCGMOS – GFS MOS DCHRCN – TPC forecast advisory DCISIG – International SIGMET DCLSFC – Surface land synoptic


Current Decoders

DCMETR – Hourly surface observations DCMSFC – Surface marine synoptic DCNCON – Non-convective SIGMET DCNMOS – NGM MOS DCRDF – Regional Digital Forecast DCSCD – Supplemental Climatological DCSVRL – Severe local storm DCTAF – Terminal aerodrome forecast DCUAIR – Upper air soundings DCWARN – Local WFO warnings


Current Decoders

DCWCN – Watch county notification DCWCP – SPC Watch corner points DCWOU – SPC Watch outline DCWSTM – Winter storm reports DCWTCH – SPC watches DCXMOS – Extended GFS MOS

NAGRIB, NAGRIB2 – GRIB NAMSND – BUFR soundings


GEMPAK Data Files

GEMPAK Data Management Files Applications do not access the data management functions directly Applications use the interface routines for the type of data being

accessed Grid (GD) programs use the GD access functions Surface (SF) programs use the SF access functions Upper Air (SN for Sounding) programs use the SN access functions

Clean interfaces That is, all information is passed in the calling sequences of the

routines, no information is shared between levels through global variables

The data management library does not interpret the data stored in the files

The data access routines interpret the information for the data type


GEMPAK Data Files

Responsibilities of the data management functions Read data from the files Write data to the files Data packing

Set up at file creation Packing, based on a table, into as few integers as necessary DM handles conversion from Real values to packed integers, and vice

versa Searches through the data headers Machine type conversions

Differences in workstations and operating systems is isolated to the DM routines

Any necessary byte swapping is handled by the DM routines only


GEMPAK Data Files

Label:

Version number, number of rows, columns and parts, file type, missing value, machine type (all set at file creation)

File Info:

File headers, free data space, part and header names, packing info for SF and SN, navigation for GD files

Row and Column Headers:

Store “key” information, Keys are used for searching

Data Pointers

Data Records

Data Management File Structure


GEMPAK Data Files


GEMPAK Data Files

Missing Data Value For packed data, the missing value has all bits turned on

Min: 0; Max: 3; Interval: 1 Offset: 0; Scale: 0; Bits: 3 Valid data values: 0, 1, 2, 3, Missing

The missing value is stored in the file header at creation

Machine Type The machine type is written to the file header When reading and writing data, the value is checked against the

current machine If necessary, the value is used to determine byte order and

swapping


Vector Graphics Files

Used for interactive product generation Used to store georeferenced objects

All locations and points in the objects are latitude/longitude

Binary format fileThe format is defined in vgstruct.hThe format was created by NAWIPS designers It is NOT a standard format known outside of

the NAWIPS system



Structure of a VG fileFollows the definition of the element structure

in vgstruct.hActually a union of structuresAll individual elements are defined in their own

structuresFront, Line, Text, Symbol, Watch, GFA, etc.The structures contain the attributes for the element

and the location(s) used to plot on a map



For exampletypedef struct frontinfo { int numpts; int fcode; int fpipsz; int fpipst; int fpipdr; int fwidth; char frtlbl[4];} FrontInfo;

typedef struct fronttype { FrontInfo info; float latlon[MAXPTS*2];} FrontType;

typedef struct lineinfo { int numpts; int lintyp; int lthw; int width; int lwhw;} LineInfo;

typedef struct linetype { LineInfo info; float latlon[MAXPTS*2];} LineType;

typedef struct fileheadtype { char version[128]; char notes[256];} FileHeadType;



For exampletypedef struct vg_hdrstruct{ char delete; char vg_type; char vg_class; signed char filled; char closed; char smooth; /* smooth level of VG elements */ char version; /* version number of VG elements */ char grptyp; int grpnum; int maj_col; int min_col; int recsz; float range_min_lat; float range_min_lon; float range_max_lat; float range_max_lon; } VG_HdrStruct;



For exampletypedef struct vg_dbstruct { VG_HdrStruct hdr; union { FileHeadType fhed; FrontType frt; LineType lin; } elem;} VG_DBStruct;


Product Generation

Challenges of manual product generation Initial drawing of elementsSetting element attributesSubsequent editing of the elements or the

attributesEfficient file formatMinimize element search times


Day 2 - Afternoon

Test Programs Common Debug Tools Utilities Points of Contact Access to SIB Systems Upcoming and Long-term Projects


Test Programs

Many GEMLIB and CGEMLIB libraries have test programs

The test programs were created to test the individual functionality of each routine

The goal is to have every function included in at least one test program

When adding a new function, make sure to add it to the appropriate test program


Test Programs

The test programs have a simple user interfaceAll input is queried from the user at the

terminalThis allows for easily testing all paths through

the codeA script can be created to run the program

many times with various input The output from the functions is printed to

the terminal


Common Debug Tools

Built-in tools for Linuxgdb is the GNU debuggerddd is a graphical front-end to gdbvalgrind is run similar to a debugger

Tracks memory usageChecks for uninitialized variablesChecks for out-of-bounds conditions for arrays

mtrace is used to trace calls to malloc and the associated memory

strace records system calls from a program


Common Debug Tools

PurifyPurchased from Rational (now part of IBM)Only for HPUXSimilar to valgrind

Checks for memory issues


Common Debug Tools

PRINT/PRINTFA very powerful toolAllows very specific control over what is being

monitoredDon’t overlook the simple solution


Utilities

Many utilities have been developed by the NAWIPS team to accomplish tasks searcher

Searches a directory tree for the requested stringCan also take action on the files found to contain the search

string whocalls

Generates a list of functions that call the requested function whatcalls

Shows a list of functions called by the specified function deliv.perl

Creates a list of functions to be deliveredThe list is suitable for inclusion on the delivery form


Utilities

Other utilities have been downloaded or are part of Linux mgdiff

Graphical difference between two filesUsed to check or reconcile changes made to files

mpagePrinting utilityCreates virtual pages so that more than one page is printed per

sheet of paper


Points of Contact

Your local NAWIPS liaison Your branch chief Steve Schotz - NAWIPS Team Leader

[email protected]

Scott Jacobs - SIB Technical [email protected]

NAWIPS [email protected] tags on the subject line to differentiate issues

BUG, DEV-QUESTION, USER-QUESTION


Access to SIB Systems

Currently, one system at each center can access HP1

New accounts have been made for each center to log into SIB systemsFirst, log into HP1, then you can log into other

platformsBy using these accounts, you will have access

to all SIB utilities These accounts will access the development

version of NAWIPS (nawdev)



Current platformsHPUX - hp1AIX - ibm2 IRIX - sgi82Linux

RHEL 3 (kernel 2.4el) - lnx64RHEL 4 (kernel 2.6el) - lnx365



New accountsdevawcdevcpcdevhpcdevopcdevspcdevtpc

Will provide passwords


Upcoming and Long-term Tasks

Grid Toolkit Graphical Object Toolkit


Grid Toolkit Topics

Objectives Benefits Stakeholders Risks Strategy FY06 Activities Preliminary FY07+ Plans


Project Objective

Develop a common grid calculator toolkit/service that can be accessed by any hydrometeorological application.


Benefits

The toolkit will: Contain a clean API and shareable architecture Be grid size independent Be independent of data format and data access

Enable a distributed data access paradigmEnable access to GRIB files directly

Allow the entire NWS to benefit from NCEP expertise in model grid algorithms Include all current GEMPAK ‘on the fly’ diagnostic

capabilities


Stakeholders

NCEP Service Centers NCO EMC AWIPS Unidata


Risks

N-AWIPS senior software analysts availability

Other NCEP requirement priorities Uncertain LOE for later phases Uncertainty in API requirements beyond N-

AWIPS


Strategy

Overlapping multi-phase – Initial goal of integration into N-AWIPS Analysis Design Development

Test and integrate after each phase in N-AWIPS initially

Plan for eventual integration into AWIPS-Next


FY06 Activities

Remove grid size limitation for calculationsAnalyze codeRewrite grid diagnostic libraries in c

~ 400 diagnostic library routines in 7 librariesRegression test scripts writtenAdd dynamic memory management for internal

grids Integrate into operational deadline – target

5.9.4 (August 2006)


Preliminary FY07+ Schedule

FY06 –FY07 High-Level activities API definition to decouple grid calculator from

GEMPAK grid file structureApplication interfaceData access interface

Design APIs Develop test plans Code APIs Modify N-AWIPS applications to use new APIs Integrate and test in N-AWIPS


Graphical Object Toolkit Topics

Background Objectives Benefits Stakeholders Risks Strategy FY06 Activities Preliminary FY07+ Plans


Background

N-AWIPS Vector Graphics (VG) files are key component of NCEP graphical product generation

VG files attributes Contain meta information and earth locations for large suite of

meteorological objects and graphical products Fronts, jets, meteorological symbols, clouds, Sigmets, jets, watches,

etc. VG files are used as basis to generate other product formats

Text Products, GIF, BUFR, etc. VG files are currently in-house developed binary file format


Project Objective

Redesign the VG file format to use Extensible Mark Up Language (XML) based format so that NCEP graphical products can be more readily ingested by external systems.

Develop a toolkit that provides a simple set of APIs so that applications beyond the NAWIPS suite can easily access and store meteorological graphical objects.


Benefits

NCEP graphical products more easily converted to other formats for use by outside users GIS systems proprietary formats Possible web displays

SVG (a W3C standard)X3d (a W3C standard)

Facilitate conversion to BUFR Greatly facilitate conversion to text products


Benefits- Continued

XML file format is a standard Internationally recognized World Wide Web

Consortium (W3C) standard U.S. Government accepted standard Platform independent

Extensible solution New attributes/objects easily incorporated Backward file compatibility more easily managed


Benefits- Continued

Toolkit provides APIs so that outside groups benefit from N-AWIPS team expertise in managing meteorological objectsEnable intelligent retrieval and storage of

meteorological objects and attributesBy object type, location or group, e.g, retrieve all

cold fronts or closest cold front to a location Interprets objects into XML files and vice versa


Stakeholders

NCEP Service Centers NCO AWIPS Unidata External Users and Systems


Risks

N-AWIPS senior software analysts availability Other NCEP requirement priorities Staged implementation into NMAP is difficult;

risk mitigation strategy includes: Initial analysis and prototyping phase Development done in parallel with incremental testing

using NCEP graphical products Integrate into N-AWIPS with extensive parallel testing


FY06 Activities

Preliminary Analysis Conducted Federal Geographic Data Committee (FGDC)

specification for geospatial meta data (file header) investigated

Preliminary XML schema analysis conducted XML libraries and tools installed XML training conducted for N-AWIPS team Project suspended due to resource limitations

XML approach is being used for components of GFA task


Preliminary FY07+ Plans

Develop XML schema for VG files Possibly adopt or extend Geography Mark Up

Language (GML) W3C XML specification Define toolkit interface Prototype toolkit and test using NCEP graphical

products Convert N-AWIPS to use toolkit

Perform parallel testing

Implement toolkit in N-AWIPS