Community infrastructure for building and coupling high performance climate, weather, and coastal models

Cecelia DeLuca

NOAA / CIRESUniversity of Colorado, Boulder

cecelia.deluca@noaa.govESMF website: www.earthsystemmodeling.org

Building Community Codes for Effective Scientific Research on HPC Platforms

September 6-7, 2012

Outline

• Vision and Context• Technical Overview• User Community• Values, Processes and Governance• Future

Vision

• Earth system models that can be built, assembled and reconfigured easily, using shared toolkits and standard interfaces.

• A growing pool of Earth system modeling components that, through their broad distribution and ability to interoperate, promotes the rapid transfer of knowledge.

• Earth system modelers who are able to work more productively, focusing on science rather than technical details.

• An Earth system modeling community with cost-effective, shared infrastructure development and many new opportunities for scientific collaboration.

• Accelerated scientific discovery and improved predictive capability through the social and technical influence of ESMF.

Computational Context• Teams of specialists, often at different sites, contribute scientific or

computational components to an overall modeling system• Components may be at multiple levels: individual physical processes (e.g.

atmospheric chemistry), physical realms (e.g. atmosphere, ocean), and members of same or multi-model ensembles (e.g. “MIP” experiments)

• Components contributed from multiple teams must be coupled together, often requiring transformations of data in the process(e.g. grid remapping and interpolation, merging, redistribution)

• Transformations are most frequently 2D data, but 3D is becoming more common• There is an increasing need for cross-disciplinary and inter-framework coupling

for climate impacts• Running on tens of thousands of processors is fairly routine; utilizing

hundreds of thousands of processors or GPUs is less common• Modelers will tolerate virtually no framework overhead and seek fault

tolerance and bit reproducibility• Provenance collection is increasingly important for climate simulations

Architecture

Low Level Utilities

Fields and Grids Layer

Model Layer

Components LayerGridded ComponentsCoupler Components

ESMF Infrastructure

User Code

ESMF Superstructure

MPI, NetCDF, …External Libraries

• The Earth System Modeling Framework (ESMF) provides a component architecture or superstructure for assembling geophysical components into applications.

• ESMF provides an infrastructure that modelers use to– Generate and apply

interpolation weights– Handle metadata, time

management, I/O and communications, and other common functions

The ESMF distribution does not include scientific models

Components• ESMF is based on the idea of components – sections of code that are

wrapped in standard interfaces • Components can be arranged hierarchically, helping to organize the

structure of complex models• Different modeling groups may create different kinds or levels of

componentsGEOS-5

surface fvcore gravity_wave_drag

history agcm

dynamics physics

chemistry moist_processes radiation turbulence

infrared solar lake land_ice data_ocean land

vegetation catchment

coupler

coupler coupler

coupler

coupler

coupler

coupler

Some of the ESMF components in the GEOS-5 atmospheric GCM

Applications of information

layer

ESMF as anInformation Layer

Native model data structures

Standard data structures

Standard metadata

• Parallel generation and application of interpolation weights• Run-time compliance checking of metadata and time behavior• Fast parallel I/O• Redistribution and other parallel communications• Automated documentation of models and simulations • Ability to run components in workflows and as web services

Field Grid ClockComponent

Attributes: CF conventions, ISO standards, METAFOR Common Information Model

Structured model information stored in ESMF wrappers

User data is referenced or copied into ESMF structures

modulesfields

gridstimekeeping

ESMF data structures

Standard Interfaces

• All ESMF components have the same three standard methods:

– Initialize– Run– Finalize

• Each standard method has the same simple interface:

call ESMF_GridCompRun (myComp, importState, exportState, clock, …)

Where:myComp points to the componentimportState is a structure containing input fieldsexportState is a structure containing output fieldsclock contains timestepping information

Steps to adopting ESMF• Divide the application into components

(without ESMF)• Copy or reference component input and

output data into ESMF data structures• Register components with ESMF• Set up ESMF couplers for data exchange

• Interfaces are wrappers and can often be set up in a non-intrusive way

Data Structures

• ESMF represents data as Fields that can be built on four discretization types:– Logically rectangular grids, which may be connected at the

edges – Unstructured meshes– Observational data streams– Exchange grids – grids that are the union of the grids of

components being coupled, and reference the data on the original grids

• ESMF can transform data among these representations, using an underlying finite element unstructured mesh engine

Component Overhead

• Representation of the overhead for ESMF wrapped native CCSM4 component

• For this example, ESMF wrapping required NO code changes to scientific modules

• No significant performance overhead

(< 3% is typical)

• Few code changes for codes that are modular

• Platform: IBM Power 575, bluefire, at NCAR• Model: Community Climate System Model (CCSM)• Versions: CCSM_4_0_0_beta42 and

ESMF_5_0_0_beta_snapshot_01 • Resolution: 1.25 degree x 0.9 degree global grid

with 17 vertical levels for both the atmospheric and land model, i.e. 288x192x17 grid. The data resolution for the ocean model is 320x384x60.

ESMF RegriddingESMF offers extremely fast parallel regridding with many options (regional, global; bilinear, higher order, first order conservative methods; logically rectangular grids and unstructured meshes; pole options; 2D or 3D; invocation as offline application or during model run)Summary of features:http://www.earthsystemmodeling.org/esmf_releases/non_public/ESMF_5_3_0/esmf_5_3_0_regridding_status.htm

HOMME Cubed Sphere Grid with PentagonsCourtesy Mark Taylor of Sandia

FIM Unstructured Grid Regional Grid

ESMF supported grids

IMPACT:“use of the parallel ESMF offline regridding capability has reduced the time it takes to create CLM surface datasets from hours to minutes” - Mariana Vertenstein, NCAR

• ESMF Offline RegridWeightGen Application:– Separate application that is built as part of ESMF, can be used independently– Application generates a netCDF weight file from two netCDF grid files– Supports SCRIP, GRIDSPEC, UGRID, and custom ESMF unstructured format

mpirun –np 32 ESMF_RegridWeightGen –s src_grid.nc –d dst_grid.nc –m bilinear –w weights.nc

• Regridding during model execution: – ESMF library subroutine calls which do interpolation during model run– Can get weights or feed directly into ESMF parallel sparse matrix multiply– Can be used without ESMF components

call ESMF_FieldRegridStore(srcField=src, dstField=dst, regridMethod=ESMF_REGRID_METHOD_BILINEAR, routehandle=rh)

call ESMF_FieldRegrid(srcField=src, dstField=dst, routehandle=rh)

Regridding Modes

04/24/23

ESMP – Python Interface● Flexible way to use ESMF parallel regridding● Separate download:

http://esmfcontrib.cvs.sourceforge.net/viewvc/esmfcontrib/python/ESMP/● Requirements:– Python– Numpy– Ctypes● Limited platform support: Linux/Darwin, GCC(g++/gfortran), OpenMPI● Data type: ESMP_Field● Grid types:– Single-tile 2D logically rectangular type: ESMP_Grid– Unstructured type: ESMP_Mesh● Support for all ESMF interpolation options

ESMF Web Services for Climate Impacts

GOAL: Develop a two-way coupled, distributed, service-based modeling system comprised of an atmospheric climate model and a hydrological model, utilizing standard component interfaces from each domain.IDEA: Bring the climate model to local applications

•Two way technical coupling completed during 2012 using CAM (with active land) and SWAT (Soil Water Assessment Tool)•Utilized switch in CAM’s ESMF component wrapper that enables web service interface•Coupled system is run using the OpenMI configuration editor, a web service driver developed in the hydrological community•Next steps: using WRF within the CESM framework and an updated hydrological model

Summary of Features• Components with multiple coupling and execution modes for flexibility,

including a web service execution mode• Fast parallel remapping with many features• Core methods are scalable to tens of thousands of processors• Supports hybrid (threaded/distributed) programming for optimal

performance on many computer architectures; works with codes that use OpenMP and OpenACC

• Time management utility with many calendars, forward/reverse time operations, alarms, and other features

• Metadata utility that enables comprehensive, standard metadata to be written out in standard formats

• Runs on 30+ platform/compiler combinations, exhaustive nightly regression test suite (4500+ tests) and documentation

• Couples Fortran or C-based model components• Open source license

Major UsersESMF Components:• NOAA National Weather Service operational weather models

(GFS, Global Ensemble, NEMS)• NASA atmospheric general circulation model GEOS-5• Navy and related atmospheric, ocean and coastal research and

operational models – COAMPS, NOGAPS, HYCOM, WaveWatch, others

• Hydrological modelers at Delft hydraulics, space weather modelers at NCAR and NOAA

ESMF Regridding and Python Libraries• NCAR/DOE Community Earth System Model (CESM)• Analysis and visualization packages: NCAR Command Language,

Ultrascale Visualization - Climate Data Analysis Tools (UV-CDAT), PyFerret users

• Community Surface Dynamics Modeling System

Usage Metrics• Updated ESMF component listing at:

http://www.earthsystemmodeling.org/components/• Includes 85 components with ESMF interfaces, 12 coupled cross-

agency modeling systems in space weather, climate, weather, hydrology, and coastal prediction, for operational and research use

• About 4500 registered downloads

Values and Principles• Community driven development and community ownership• Openness of project processes, management, code and information• Correctness• Commitment to a globally distributed and diverse development and

customer base• Simplicity• Efficiency• User engagement• Environmental stewardship

Web link for detail: http://www.esmf.ucar.edu/about_us/values.shtml

Agile Development vsCommunity Values

In general, Agile processes promote to community development.However, there are areas of potential conflict:

Agile emphasis on co-location: “most agile methodologies or approaches assume that the team is located in a single team room” (from Miller, Dist. Agile Development at Microsoft, 2008)

vsESMF emphasis on distributed co-development

Agile emphasis on single product ownervs

ESMF emphasis on multi-party ownership

Making DistributedCo-Development Work

Hinges on asynchronous, all-to-all communication patterns: everybody must have information

• Archived email list where all development correspondence gets cc’d• Minutes for all telecons• Web browsable repositories (main and contributions), mail summary

on check-ins• Daily, publicly archived test results• Monthly archived metrics• Public archived trackers (bugs, feature requests, support requests,

etc.)

Discouraged: IMing, one-to-one correspondence or calls – the medium matters

Change Review Board

• CRB established as a vehicle for shared ownership through user task prioritization and release content decisions

• Consists of technical leads from key user communities• Not led by the development team!• Sets the schedule and expectations for future functionality

enhancements in ESMF internal and public distributions– Based on broad user community and stakeholder input– Constrained by available developer resources– Updated quarterly to reflect current realities

• CRB reviews releases after delivery for adherence to release plan

Governance HighlightsManagement of ESMF required governance that recognized

social and cultural factors as well as technical factorsMain practical objectives of governance:• Enabling stakeholders to fight and criticize in a civilized,

contained, constructive way• Enabling people to make priority decisions based on

resource realitiesObservations:• Sometimes just getting everyone equally dissatisfied and

ready to move on is a victory• Thorough, informed criticism is the most useful input a

project can get• Governance changes and evolves over the life span of a

project

Governance Functions• Prioritize development tasks in a manner acceptable to major stakeholders

and the broader community, and define development schedules based on realistic assessments of resource constraints (CRB)

• Deliver a product that meets the needs of critical applications, including adequate and correct functionality, satisfactory performance and memory use, ... (Core)

• Support users via prompt responses to questions, training classes, minimal code changes for adoption, thorough documentation, ... (Core)

• Encourage community participation in design and implementation decisions frequently throughout the development cycle (JST)

• Leverage contributions of software from the community when possible (JST)• Create frank and constructive mechanisms for feedback (Adv. Board)• Enable stakeholders to modify the organizational structure as required

(Exec. Board)• Coordinate and communicate at many levels in order to create a

knowledgeable and supportive network that includes developers, technical management, institutional management, and program management (IAWG and other bodies)

GovernanceExecutive BoardStrategic DirectionOrganizational ChangesBoard Appointments

Interagency Working GroupStakeholder LiaisonProgrammatic Assessment & Feedback

Advisory BoardExternal Projects CoordinationGeneral Guidance & Evaluation

Functionality Change Requests

Joint Specification TeamRequirements DefinitionDesign and Code ReviewsExternal Code Contributions

Implementation Schedule

Resource Constraints

Collaborative DesignBeta Testing

Working Project

ExecutiveManagement

Reporting

Reporting

weekly

Core Development TeamProject ManagementSoftware DevelopmentTesting & MaintenanceDistribution & User Supportdaily

annually

Change Review BoardDevelopment PrioritiesRelease Review & Approval

quarterly

EvolutionPhase 1: 2002-2005NASA’s Earth Science Technology Office ran a solicitation to develop an Earth System Modeling Framework (ESMF).A multi-agency collaboration (NASA/NSF/DOE/NOAA) won the award. The core development team was located at NCAR.A prototype ESMF software package (version 2r) demonstrated feasibility.

Phase 2: 2005-2010New sponsors included Department of Defense and NOAA.A multi-agency governance plan including the CRB was created:http://www.earthsystemmodeling.org/management/paper_1004_projectplan.pdfMany new applications and requirements were brought into the project, motivating a complete redesign of framework data structures (version 3r).

Phase 3: 2010-2015 (and beyond)The core development team moved to NOAA/CIRES for closer alignment with federal models.Basic framework development completed with version 5r (ports, bugs, feature requests, user support etc. still require resources).The focus is on increasing adoption and creating a community of interoperable codes.

Technical Evolution:National Unified Operational

Prediction Capability (NUOPC)

• ESMF allows for many levels of components, types of components, and types of connections

• In order to achieve greater interoperability, usage and content conventions and component templates are needed

• A tri-agency collaboration (NOAA, Navy, Air Force) is building a “NUOPC Layer” that constrains how ESMF is used, and introduces metadata and other content standards, along with inheritable templates for different usage scenarios

• A production version of the NUOPC Layer is scheduled for delivery at the end of 2012

CoG Collaboration Environment

Right side: ServicesCentral navigation: Templated content

Central Section: Freeform content

Left side: Auto generated

navigation of project

freeform content

The CoG environment exposes and collates the information needed for distributed, multi-project development, including project repositories, trackers, and governance processes. It does this in an environment that’s linked to data search, metadata, and visualization services and set up to enable component comparisons.

DCMIP on CoGAtmospheric Dynamical Core Model Intercomparison Project

http://www.earthsystemcog.org/projects/dcmip-2012/

Planned MIPs:2013 Downscaling 2014Atm-surface hydrology2014 Frameworks

04/24/23

Future

Governance and process largely follow established patternsActive development● Python API redesign and addition of features● Preparation for NUOPC Layer major release● Web services for climate impacts● Regridding options – higher order conservative , filling in gaps● Advanced fault-tolerance● Performance optimizations and GPUs● CoG collaboration environment

ESMF and related development is supported by NASA, NOAA, NSF, and theDepartment of Defense.