gne: g lobal n ews modeling e nvironment

GNE: Global NEWS Modeling Environment

What it is / what it’s notHow it works

Components, samplesInstallation

NEWS 2 Implementation

Emilio Mayorga May [email protected] http://www.marine.rutgers.edu/globalnews/GNE

GNE Training Session 1

Content last edited August 28, 2008

Generic GNE vs. MA Scenarios Specifics

• Covering all details of Millennium Assessment (MA) Scenarios implementation is not an objective of these training sessions

• But hands-on sessions and demonstrations will be based on MA 2000 (c00) scenario

• Will use MA implementation to provide concrete examples

GNE Training Goals• Introduce & describe GNE

– Design goals, main features, code files

• Describe a standard GNE model run– Major components, files (code, configuration, and input/output

data), organization and conventions

• Describe input pre-processing and output post-processing

• Enable “model users” to set up and execute model scenario experiments– Modify input data sources used (especially for MA scenarios)– No Python coding needed

Refer to GNE User Manual for more complete details. Sessions provide more interactive training and demonstration, while manual includes extensive details about syntax, usage, installation, etc.

Training Sessions

• Session 1. Introduction and Broad OverviewThis Presentation

• Session 2. Running NEWS models with pre-existing, basin-aggregated input tablesInteractive Session, in 2 parts (2 sub-sessions)– Part 1. Model run organization, configuration files, conventions– Part 2. Model execution; changing a model run (“experiments”)

• Session 3. GIS pre-processing and model input/output post-processing

Interactive Session

Session 1 Content, Goals• Overall GNE design goals, highlights• Why Python?• Three GNE User Types

– Training focus on “Model Users” (no Python coding needed)

• GNE vs. end-to-end Global NEWS Run– What is covered by GNE vs. what is not

• Major Components– Code, configuration files, input/output files, file organization and

conventions

• GNE Model Run folder & files• GNE Model Run• GIS Pre-Processing (GNE vs. other)• Post-Processing (GNE vs. other)• Python Installation and Base GNE Setup

Overall GNE design goals, highlights

1. Dissolved models merged into a unified framework, rather than isolated sub-models; particulate models remain separate.

2. Focus on basin-scale calculations.3. For users not needing to modify model equations, most

interactions are handled via structured text configuration files. • Together with model equations in code, these files also serve as documentation

of input files used and how output was created.

4. Complete model run can be maintained in a folder structure that includes all input data, model code, constants used, configuration files, and output tables.

5. Python/NumPy. Fairly easy to learn. Model equation code implemented in an intuitive, simple array syntax.

• More complex GNE code handling input/output and overall model coordination hidden away and most users never have to see it.

6. Model run time (including GIS pre-processing) is short or trivial.

Why Python?• Free, open source, high-level but general programming

language with large user base in all fields of computing. Free = available to everyone.

• Runs on all major operating systems.• Rapidly growing acceptance in science applications,

including modeling & GIS.• Very clean, simplified syntax that makes it easy to learn,

read existing code, and become productive quickly.• NumPy “add-on” used to code the model equations;

intuitive, simple array syntax similar to Matlab.• Large library of default and optional add-ons contributed

by users, for all sorts of applications.• Preferred language chosen by ESRI for data processing

automation in ArcGIS (ArcGIS “GeoProcessing”).

Three GNE User Types

1. Model Users. Setup or change a model run using a set of text configuration files; organize input and output files in a regular, structured fashion.

Python skills not required. This training is focused on Model

Users!

2. Model Developers. Focus on translating model equations into Python GNE code using simple code tools for variable management provided by the GNE framework. Largely isolated from the “behind-the-scenes”, low-level GNE functionality that handles file read-write and basic error checking. Should be fairly comfortable programming with Python & NumPy.

3. GNE Developers. Modify any aspect of GNE functionality, including model equations.

Should be very comfortable programming with Python & NumPy.

GNE vs. end-to-end Global NEWS Run

MNPDiffuse sources

MNPPoint sources

UNHHydrology

“Raw”, HeterogeneousGIS Source files

Diffusesources

Pre-processed, homogenizedGIS Input files

Pointsources

Hydrology

Manual steps,Ad-hoc automation Direct

Global NEWSinputs

Basin-scaleInputs tables

Global NEWSoutputs

Model Outputs

GNE

Complicated variables other than basin means/sums.Eg, dam retention, FQrem, lithology class

Glo

bal N

EW

S

mod

el(s

)

GN

E G

IS

pre-

proc

essi

ngSummaries,Analysis,

Maps

Mass fluxes

Other post-processing

GNE post-processing

Major Components• Overall folder organization

– Model run code, configuration, input/output files, etc.

• Python Code– GNE “core” (file read/write, etc.)– Master controller (globalnews.py).– Science code (Global NEWS model): dissolved.py &

particulate.py– Loads post-processor (postprocess.py)– Other, optional supporting files (batch runs, etc.)

• Configuration files (text/ASCII)– Base folder paths (gensetup.cfg)– Dissolved calibration constants (constants.cfg)– GIS basin inputs pre-processing (gis2tbls.cfg)– Global NEWS model run (vars.cfg)– Loads post-processor (also vars.cfg)

GNE Model Run Folder & Files

Files in base folderBase folder and

sub-folders

Optional files(also in base folder,

or in OTHERFILES)

Also, documentation files.

GNE Model Run Folder & Files:Python code

Files in base folderBase folder and

sub-folders

GNE Core code

Science (model) code

Master controller

Optional GISpre-processing add-on

Loadspost-processor

GNE Model Run Folder & Files:Configuration filesFiles in base folder

GIS pre-processing

Model run andloads post-processing

Dissolvedcalibration constants

Base folder paths

GNE Model Run Folder & Files:Input and output files

Base folder andsub-folders

Basin-scale model outputand post-processed tables

Pre-processedbasin-scale input tables

Homogenizedraster GIS inputs

Basins, cell areas, andother basic input data

(raster GIS and basin-scale tables)

MNPDiffuse sources

MNPPoint sources

UNHHydrology


Diffusesources


Pointsources

Hydrology


Global NEWSinputs


Global NEWSoutputs

Model Outputs

GNE


Glo

bal N

EW

S

mod

el(s

)

GN

E G

IS

pre-

proc

essi


Maps

Mass fluxes


GNE post-processing

GNE Model Run

Sample GNE Model Run Configuration File

vars.cfg

Sample GNE Model Code

Fromdissolved.py

calculateFEriv

MNPDiffuse sources

MNPPoint sources

UNHHydrology


Diffusesources


Pointsources

Hydrology


Global NEWSinputs


Global NEWSoutputs

Model Outputs

GNE


Glo

bal N

EW

S

mod

el(s

)

GN

E G

IS

pre-

proc

essi


Maps

Mass fluxes


GNE post-processing

GNE GIS Pre-Processing

MNPDiffuse sources

MNPPoint sources

UNHHydrology


Diffusesources


Pointsources

Hydrology


Global NEWSinputs


Global NEWSoutputs

Model Outputs

GNE


Glo

bal N

EW

S

mod

el(s

)

GN

E G

IS

pre-

proc

essi


Maps

Mass fluxes


GNE post-processing

Other GIS Pre-Processing

Mostly automated for MA scenarios,but in “ad-hoc” schemes,

different software, etc.

MNPDiffuse sources

MNPPoint sources

UNHHydrology


Diffusesources


Pointsources

Hydrology


Global NEWSinputs


Global NEWSoutputs

Model Outputs

GNE


Glo

bal N

EW

S

mod

el(s

)

GN

E G

IS

pre-

proc

essi


Maps

Mass fluxes


GNE post-processing

GNE “Loads” Post-Processing

Calculates basin input andoutput variables as loads

MNPDiffuse sources

MNPPoint sources

UNHHydrology


Diffusesources


Pointsources

Hydrology


Global NEWSinputs


Global NEWSoutputs

Model Outputs

GNE


Glo

bal N

EW

S

mod

el(s

)

GN

E G

IS

pre-

proc

essi


Maps

Mass fluxes


GNE post-processing

Other Post-Processing

Python Installation and Base GNE Setup• Software Installation and Configuration

– Is ArcGIS Installed? Potential conflicts with ArcGIS’ Python

– Python (if ArcGIS not installed, or very old)

– NumPy add-on

– ArcGIS needed for GNE GIS pre-processing

– (Other add-ons and custom scripts for additional post-processing: maps, regional summaries, etc.)

• GNE– Copy GNE files (code, inputs, configuration files, folder structure).

Unpack training zip file

– Avoid blank spaces or characters other than underscores and alphanumeric characters anywhere in the folder path!

– Testing

• See User Manual for complete installation details, links to online Python/NumPy resources

Next, in Session 2 …• Session 1. Introduction and Broad Overview

This Presentation

• Session 2. Running NEWS models with pre-existing, basin-aggregated input tablesInteractive Session, in 2 parts (2 sub-sessions)– Part 1. Model run organization, configuration files, conventions– Part 2. Model execution; changing a model run (“experiments”)

We will change and execute a few model runs

• Session 3. GIS pre-processing and model input/output post-processing

Interactive Session

gne: g lobal n ews modeling e nvironment

Documents

model code

model equations

conventionsgne model

model execution

model equation code

complete model run

generic gne

files code