Extension of the ESMFfor Space Weather

Cecelia DeLuca cdeluca@ucar.du SWW

April 7, 2005

http://www.esmf.ucar.edu

NSIPP Seasonal Forecast

NCAR/LANL CCSM NCEP ForecastGFDL FMS Suite

MITgcmNASA GSFC GSI

Outline

• Overview• Application Integration Projects• ESMF Status• Extension to Space Weather

ESMF Goals

1. Increase scientific productivity by making model components much easier to build, combine, and exchange, and by enabling modelers to take full advantage of high-end computers.

2. Promote new scientific opportunities and services through community building and increased interoperability of codes (impacts in collaboration, code validation and tuning, teaching, migration from research to operations).

What is ESMF?1. ESMF provides tools for turning model codes into components with standard

interfaces and standard drivers2. ESMF provides data structures and common utilities that components use for

routine services such as data communications, regridding, time management and message logging

ESMF InfrastructureData Classes: Bundle, Field, Grid, Array

Utility Classes: Clock, LogErr, DELayout, Machine

ESMF SuperstructureAppDriver

Component Classes: GridComp, CplComp, State

User Code

GEOS-5

surface fvcore gravity_wave_drag

history agcm

dynamics physics

chemistry moist_processes radiation turbulence

infrared solar lake land_ice data_ocean land

vegetation catchment

coupler

coupler coupler

coupler

coupler

coupler

coupler

• Each box is an ESMF component• Every component has standard methods and interfaces to facilitate exchanges• New components can be added to the application systematically• Data in and out of components are state types that contain fields, bundles of fields • Coupling tools include regridding and redistribution methods

Application Example: GEOS-5 AGCM

ESMF Community◦ NSF National Center for Atmospheric Research ◦ NOAA Geophysical Fluid Dynamics Laboratory ◦ NOAA National Centers for Environmental Prediction◦ NASA Goddard Global Modeling and Assimilation Office◦ NASA Goddard Institute for Space Studies ◦ NASA Jet Propulsion Laboratory◦ NASA Goddard Land Information Systems project ◦ DOD Naval Research Laboratory ◦ DOD Air Force Weather Agency ◦ DOD Army Engineer Research and Development Center ◦ DOE Los Alamos National Laboratory ◦ DOE Argonne National Laboratory ◦ University of Michigan◦ Princeton University◦ Massachusetts Institute of Technology ◦ UCLA◦ Center for Ocean-Land-Atmosphere Studies ◦ Programme for Integrated Earth System Modeling (PRISM) ◦ Common Component Architecture (CCA)

ESMF Impacts

ESMF impacts a very broad set of research and operational areas that require high performance, multi-component modeling and data assimilation systems, including: • Climate prediction• Weather forecasting• Seasonal prediction• Basic Earth and planetary system research at various time and spatial scales• Emergency response• Ecosystem modeling• Battlespace simulation and integrated Earth/space forecasting• Space weather (through coordination with related space weather frameworks)• Other HPC domains, through migration of non-domain specific capabilities from ESMF – facilitated by ESMF interoperability with generic frameworks such as CCA

Infrastructure Timeline: ESMF Path

• ESMF common modeling infrastructure (~5 year)Technical foundation that allows for organized and exchangeable codes

• Set of modeling codes that are based on ESMF (~5 year)Community pool of interoperable science components with which to assemble applications

• Multi-agency and international organization (~5 year) Organizational foundation on which to establish collaborations and set priorities

• Standards for model and data description (~10 year)Prerequisite for an advanced modeling and collaboration environment that includes knowledge management

• ESMF part of an established end-to-end community-based modeling environment (~20 year)An Earth System Modeling Environment (ESME) that combines models, data, experiments, collaborative tools and information resources in a way that fosters knowledge sharing and accelerates scientific workflow

ESMF Project

Develops and manages the ESMF Product

DoD BEI Project

Generates ESMF-compliant science applications

NASA Modeling, Analysis, and Prediction Environment

Generates ESMF-compliant science applications NOAA ESMF Adoption

Project

Generates ESMF-compliant science applications

Application Integration Projects and Development Plug-Ins from the DoD, NASA, NOAA, and elsewhere interact with the ESMF Project. Areas of overlap are contributions of funds and code to the core development effort, and participation in ESMF development and management.

Other Application

Integration Projects and Development Plug-Ins

CCSM, WRF, university applications, more ….

ESMF and Application Integration Projects

NASA CAN ESMF Project DescriptionPRODUCTS: • Core framework: Software for coupling geophysical components and utilities for building

components • Applications: Deployment of the ESMF in 15 of the nation’s leading climate and weather

models, assembly of 8 new science-motivated applicationsMETRICS:

RESOURCES and TIMELINE: $9.8M from NASA over 3 years, starting FY02

Reuse Interoperability Ease of Adoption Performance

15 applications use ESMF component coupling services and 3+ utilities

8 new applications comprised of never-before coupled components

2 codes adopt ESMF with < 2% lines of code changed, or within 120 FTE-hours

No more than 10% overhead in time to solution, no degradation in scaling

NASA CAN Adoption Status0 1 2 3 4 5 6 7 8

EVA Test Atmosphere

EVA Test Coupled Atm/Ocn

GFDL B-Grid Atm

GFDL Spectral Atm

GFDL MOM4 Ocean

MITgcm Coupled Atm/Ocn

MITgcm Regional Ocn

NCAR CAM / fvCAM

NCAR CLM

LANL POP

LANL CICE

CCSM

GEOS-5 Coupled Atm/Ocn

NCEP GFS

NCEP SSI

NCEP/NCAR WRF

ODAS w ith NSIPP Ocn

MITgcm Adjoint

GSI Analysis

GEOS-5 Atmosphere

Superstructure (4 = running asESMF component)

Infrastructure (number of ESMFutilities used)

adoptiondue summer 2005

DoD Battlespace Environments Institute

To develop a DoD-wide whole-earth environment which interoperates with that from other agencies:◦ Migrate core DoD models to ESMF

• Navy (e.g., NCOM, HYCOM, SWAN, COAMPS™)• Air Force (Kinematic Solar Wind and GAIM )• Army (e.g., ADCIRC, WASH123)

◦ Development of tools and applications (e.g., extend ESMF to support unstructured grids and nesting)

◦ Coupled applications: Air/ocean, air/ocean/ice, air/ocean/groundwater, air/space-weather

◦ Thorough testing, prototyping, documentation of all components◦ $11M over six years of funding, starting FY05

NASA Modeling Analysis and Prediction Program

• ESMF-based computational environment for researchers investigating climate variability and change

• New set of climate, chemistry, and related applications coming in to ESMF• NASA plan includes support for Independent Test Team• Awards to be announced April 2005

ESMF Development Status• Overall architecture is well-defined• Components and low-level communications stable• Concurrent or sequential execution, single executable• Serial or parallel execution (mpiuni stub library)• Logically rectangular grids with regular and arbitrary distributions can be represented

and regridded• On-line parallel regridding (bilinear, 1st order conservative) completed and optimized• Other parallel methods, e.g. halo, redistribution, low-level comms implemented• Utilities such as time manager, logging, and configuration manager usable and

adding features• Virtual machine with hooks for load balancing implemented• Fortran interfaces and complete documentation, some C++ interfaces

ESMF Platform Support• IBM AIX (32 and 64 bit addressing) • SGI IRIX64 (32 and 64 bit addressing) • Compaq OSF1 (64 bit addressing) • Linux Intel (32 and 64 bit addressing, with mpich and lam) • Linux PGI (32 bit addressing, with mpich) • Linux NAG (32 bit addressing, with mpich) • Linux Absoft (32 bit addressing, with mpich) • Linux Lahey (32 bit addressing, with mpich) • Mac XLF (32 bit addressing, with lam) • Mac Absoft (32 bit addressing, with lam) • SGI Altix (64 bit addressing) • Cray X1 (64 bit addressing)

These are the platforms and options that we test on nightly. Other options are provided in the ESMF build system.

Some Metrics …• Core Implementation Team at NCAR currently has

◦ 2 FTE testers◦ 1/2 FTE performance analyst ◦ 5 FTE developers◦ 1 FTE admin/web support◦ 1 managerAnticipate sustained level of staff for next 5-6 years

• Test suite currently consists of◦ ~1250 unit tests◦ ~15 system tests◦ ~40 examples

runs every night on ~12 platforms, ~ 40 configurations• ~284 ESMF interfaces implemented, ~266 fully or partially tested, ~93% fully or

partially tested – list of untested methods on ESMF website• ~150,000 SLOC• ~1000 downloads

Open Source Development• Open source license (GPL) • Open source environment (SourceForge)• Open repositories: web-browsable CVS repositories accessible from the

ESMF website http://www.esmf.ucar.edu◦ for source code◦ for contributions (currently porting contributions and performance testing)

• Open development priorities and schedule: priorities set by users (previously during meetings and telecons, transition to new Change Review Board), web-browsable task lists

• Open communication: frequent telecons and mailing list discussions • Open testing: all tests are bundled with the ESMF distribution and can be run

by users• Open port status: results of nightly tests on many platforms are web-browsable• Open metrics: test coverage, lines of code, requirements status are updated

regularly and are web-browsable

ESMF Component Overhead

• Measures overhead of ESMF superstructure in NCEP Spectral Statistical Analysis (SSI), ~1% overall

• Run on NCAR IBM

• Runs done by JPL staff, confirmed by NCEP developers

ESMF Current Challenges• Process and techniques for software requirements analysis,

design and interface review with a VERY large distributed community (installation of commercial software (DOORS) for requirements archival and tracking, ties to testing and tasking)

• Transition to multi-agency organizational structure• Design strategy for advanced grids and regridding; science and

math hires to implement new grids and regridding methods• Continued improvement of development and test processes• Clear, complete, carefully edited documentation and training

program materials• New collaborations for technical and domain extension

Planned ESMF Extensions1. Looser couplings: support for multiple executable and Grid-enabled versions of

ESMF2. Support for representing, partitioning, communicating with, and regridding

unstructured grids and semi-structured grids3. Support for advanced I/O, including I/O support for both logically rectangular,

semi-structured, and unstructured grids, asynchronous I/O, checkpoint/restart, and multiple archival mechanisms (e.g. NetCDF, HDF5, binary, etc.)

4. Advanced support for data assimilation systems, including data structures for observational data and adjoints for ESMF methods

5. Support for nested, moving grids and adaptive grids 6. Support for regridding in three dimensions and between different coordinate

systems7. Advanced optimization and load balancing

Extension Process

• Meaningful agreement for Earth/space frameworks to collaborate and interoperate (i.e., includes resources, staged goals, timeline)

• Collective analysis and archival of requirements for space weather applications

• Collective and ongoing design, review, and testing• Participation in joint meetings and events (e.g. GEM, SWW,

ESMF Annual Meeting (July 20-22, Cambridge, MA)• Shared tools?