introduction to the earth system modeling framework v. balaji, gfdl [email protected] cecelia...
TRANSCRIPT
Introduction to the Earth System Modeling Framework
V. Balaji, GFDL [email protected] DeLuca, [email protected] Chris Hill, MIT [email protected]
Coastal Inundation Workshop
25 January 2006
ClimateData Assimilation
Weather
2
OUTLINE
• Overview• ESMF and the Community• Development Status• Design and Principles of ESMF• Resources
3
Motivation and Context
In climate research and NWP... increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall modeling system
In computing technology... increase in hardware and software complexity in high-performance computing, as we shift toward the use of scalable computing architectures
In software …development of first-generation frameworks, such as FMS, GEMS, CCA and WRF, that encourage software reuse and interoperability
4
What is ESMF?• ESMF provides tools for turning model codes
into components with standard interfaces and standard drivers.
• ESMF provides data structures and common utilities that components use for routine services such as data communications, regridding, time management and message logging.
ESMF InfrastructureData Classes: Bundle, Field, Grid, Array
Utility Classes: Clock, LogErr, DELayout, Machine
ESMF SuperstructureAppDriver
Component Classes: GridComp, CplComp, State
User Code
ESMF GOALS
1. Increase scientific productivity by making model components much easier to build, combine, and exchange, and by enabling modelers to take full advantage of high-end computers.
2. Promote new scientific opportunities and services through community building and increased interoperability of codes (impacts in collaboration, code validation and tuning, teaching, migration from research to operations)
5
GEOS-5
surface fvcore gravi ty_wave_drag
history agcm
dynamics physics
chemistry moist_processes radiation turbulence
infrared solar lake land_ice data_ocean land
vegetation catchment
coupler
coupler coupler
coupler
coupler
coupler
coupler
• Each box is an ESMF component• Every component has a standard interface so that it is swappable• Data in and out of components are packaged as state types with user-defined fields• New components can easily be added to the hierarchical system• Coupling tools include regridding and redistribution methods
Application Example: GEOS-5 AGCM
6
Why Should I Adopt ESMF If I Already Have a Working Model?• There is an emerging pool of other ESMF-based science components that you
will be able to interoperate with to create applications - a framework for interoperability is only as valuable as the set of groups that use it.
• It will reduce the amount of infrastructure code that you need to maintain and write, and allow you to focus more resources on science development.
• ESMF provides solutions to two of the hardest problems in model development: structuring large, multi-component applications so that they are easy to use and extend, and achieving performance portability on a wide variety of parallel architectures.
• It may be better software (better features, better performance portability, better tested, better documented and better funded into the future) than the infrastructure software that you are currently using.
• Community development and use means that the ESMF software is widely reviewed and tested, and that you can leverage contributions from other groups.
7
New ESMF-Based ProgramsFunding for Science, Adoption, and Core DevelopmentModeling, Analysis and Prediction Program for Climate Variability and ChangeSponsor: NASAPartners:University of Colorado at Boulder, University of Maryland, Duke University, NASA Goddard Space Flight Center, NASA Langley, NASA Jet Propulsion Laboratory, Georgia Institute of Technology, Portland State University, University of North Dakota, Johns Hopkins University, Goddard Institute for Space Studies, University of Wisconsin, Harvard University, moreThe NASA Modeling, Analysis and Prediction Program will develop an ESMF-based modeling and analysis environment to study climate variability and change.
Battlespace Environments InstituteSponsor: Department of DefensePartners:DoD Naval Research Laboratory, DoD Fleet Numerical, DoD Army ERDC, DoD Air Force Air Force Weather AgencyThe Battlespace Environments Institute is developing integrated Earth and space forecasting systems that use ESMF as a standard for component coupling.
Integrated Dynamics through Earth’s Atmosphere and Space Weather InitiativesSponsors: NASA, NSFPartners: University of Michigan/SWMF, Boston University/CISM, University of Maryland, NASA Goddard Space Flight Center, NOAA CIRESESMF developers are working with the University of Michigan and others to develop the capability to couple together Earth and space software components.
Spanning the Gap Between Models and Datasets:Earth System CuratorSponsor: NSFPartners:Princeton University, Georgia Institute of Technology, Massachusetts Institute of Technology, PCMDI, NOAA GFDL, NOAA PMEL, DOE ESGThe ESMF team is working with data specialists to extend and unify climate model and dataset descriptors, and to create, based on this metadata, an end-to-end knowledge environment.
8
ESMF ImpactsESMF impacts a very broad set of research and operational areas that require high performance, multi-component modeling and data assimilation systems, including: • Climate prediction• Weather forecasting• Seasonal prediction• Basic Earth and planetary system research at various time and spatial scales• Emergency response• Ecosystem modeling• Battlespace simulation and integrated Earth/space forecasting• Space weather (through coordination with related space weather frameworks)• Other HPC domains, through migration of non-domain specific capabilities from ESMF – facilitated by ESMF interoperability with generic frameworks, e.g. CCA
9
OUTLINE
• Overview• ESMF and the Community• Development Status• Design and Principles of ESMF• Resources
10
Collaborative Development• Users define development priorities via a Change Review Board• Users contribute to the framework design through public design
reviews• Users help to test the framework implementation• ~15% of ESMF source code is currently from user contributions
◦ IO from WRF◦ Resource file manager from GMAO◦ Regridding from Los Alamos◦ 3D grids from University of Michigan◦ C++ component interfaces from NRL◦ More contributions in progress
11
Open Source Development• Open source license (GPL) • Open source environment (SourceForge)• Open repositories: web-browsable CVS repositories accessible
from the ESMF website ◦ for source code◦ for contributions (currently porting contributions and
performance testing)• Open testing: 1000+ tests are bundled with the ESMF distribution
and can be run by users• Open port status: results of nightly tests on many platforms are
web-browsable• Open metrics: test coverage, lines of code, requirements status
are updated regularly and are web-browsable
12
Open Source Constraints• ESMF does not allow unmoderated check-ins to its main source
CVS repository (though there is minimal check-in oversight for the contributions repository)
• ESMF has a co-located, line managed Core Team whose members are dedicated to framework implementation and support – it does not rely on volunteer labor
• ESMF actively sets priorities based on user needs and feedback• ESMF requires that contributions follow project conventions and
standards for code and documentation• ESMF schedules regular releases and meetings
The above are necessary for development to proceed at the pace desired by sponsors and users, and to provide the level of quality and customer support necessary for codes in this domain
13
Related Projects
• PRISM is an ongoing European Earth system modeling infrastructure project.
• Involves current state-of-the-art atmosphere, ocean, sea-ice, atmospheric chemistry, land-surface and ocean-biogeochemistry models.
• 22 partners: leading climate researchers and computer vendors, includes MPI, KNMI, UK Met Office, CERFACS, ECMWF, DMI.
• ESMF and PRISM are working together through a NASA MAP grant to merge frameworks and develop common conventions.
• ESMF and PRISM lead a WCRP/WMP task force on strategies for developing international common modeling infrastructure.
• CCA is creating a minimal interface and sets of tools for linking high performance components. CCA can be used to implement frameworks and standards developed in specific domains (such as ESMF).
• Collaborators include LANL, ANL, LLNL, ORNL, Sandia, University of Tennessee, and many more. There is ongoing ESMF collaboration with CCA/LANL on language interoperability.
• Working prototype demonstrating CCA/ESMF interoperability, presented at SC2003.
14
OUTLINE
• Overview• ESMF and the Community• Development Status• Design and Principles of ESMF• Resources
15
ESMF Development Status• Overall architecture well-defined and well-accepted• Components and low-level communications stable• Rectilinear grids with regular and arbitrary distributions implemented• On-line parallel regridding (bilinear, 1st order conservative) completed
and optimized• Other parallel methods, e.g. halo, redistribution, low-level comms
implemented• Utilities such as time manager, logging, and configuration manager
usable and adding features• Virtual machine with interface to shared / distributed memory
implemented, hooks for load balancing implemented
16
ESMF Distribution Summary• Fortran interfaces and complete documentation• Many C++ interfaces, no manuals yet• Serial or parallel execution (mpiuni stub library)• Sequential or concurrent execution• Single executable (SPMD) and limited multiple executable
(MPMD) support
17
ESMF Platform Support• IBM AIX (32 and 64 bit addressing) • SGI IRIX64 (32 and 64 bit addressing) • SGI Altix (64 bit addressing) • Cray X1 (64 bit addressing)• Compaq OSF1 (64 bit addressing) • Linux Intel (32 and 64 bit addressing, with mpich and lam) • Linux PGI (32 and 64 bit addressing, with mpich) • Linux NAG (32 bit addressing, with mpich) • Linux Absoft (32 bit addressing, with mpich) • Linux Lahey (32 bit addressing, with mpich) • Mac OS X with xlf (32 bit addressing, with lam)• Mac OS X with absoft (32 bit addressing, with lam)• Mac OS X with NAG (32 bit addressing, with lam)
• User-contributed g95 support
18
Some Metrics …• Test suite currently consists of
◦ ~1800 unit tests◦ ~15 system tests◦ ~35 examples
runs every night on ~12 platforms• ~291 ESMF interfaces implemented, ~278 fully or partially tested,
~95% fully or partially tested.• ~170,000 SLOC• ~1000 downloads
19
ESMF Near-Term Priorities, FY06
• Read/write interpolation weights and more flexible interfaces for regridding
• Support for general curvilinear coordinates• Reworked design and implementation of array/grid/field
interfaces and array-level communications• Grid masks and merges• Unstructured grids• Asynchronous I/O
20
Planned ESMF Extensions
1. Looser couplings: support for multiple executable and Grid-enabled versions of ESMF
2. Support for representing, partitioning, communicating with, and regridding unstructured grids and semi-structured grids
3. Support for advanced I/O, including support for asynchronous I/O, checkpoint/restart, and multiple archival mechanisms (e.g. NetCDF, HDF5, binary, etc.)
4. Support for data assimilation systems, including data structures for observational data and adjoints for ESMF methods
5. Support for nested, moving grids and adaptive grids 6. Support for regridding in three dimensions and between different
coordinate systems7. Ongoing optimization and load balancing
21
OUTLINE
• Overview• ESMF and the Community• Development Status• Design and Principles of ESMF• Resources
22
Computational Characteristicsof Weather/Climate• Mix of global transforms and local communications
• Load balancing for diurnal cycle, event (e.g. storm) tracking• Applications typically require 10s of GFLOPS,
100s of PEs – but can go to 10s of TFLOPS, 1000s of PEs• Required Unix/Linux platforms span laptop to
Earth Simulator• Multi-component applications: component
hierarchies, ensembles, and exchanges;components in multiple contexts
• Data and grid transformations between components
• Applications may be MPMD/SPMD, concurrent/sequential, combinations
• Parallelization via MPI, OpenMP, shmem, combinations• Large applications (typically 100,000+ lines of source code)
Platforms
assim
sea iceocean
landatm
physics
dycore
assim_atm
atmland
Seasonal Forecastcoupler
23
Design Strategy:Hierarchical ApplicationsSince each ESMF application is also a Gridded Component, entire ESMF applications can be nested within larger applications. This strategy can be used to systematically compose very large, multi-component codes.
GEOS-5
surface fvcore gravi ty_wave_drag
history agcm
dynamics physics
chemistry moist_processes radiation turbulence
infrared solar lake land_ice data_ocean land
vegetation catchment
coupler
coupler coupler
coupler
coupler
coupler
coupler
24
Design Strategy: ModularityGridded Components don’t have access to the internals of other Gridded Components, and don’t store any coupling information. Gridded Components pass their States to other components through their argument list.Since components are not hard-wired into particular configurations and do not carry coupling information, components can be used more easily in multiple contexts.
atm_comp
NWP application
Seasonal prediction
Standalone for basic research
25
Design Strategy: Flexibility• Users write their own drivers as well as their own Gridded Components and
Coupler Components• Users decide on their own control flow
Ocean
S eaIce
Land Coupler
A tmosphere
DATA DATA
D A T A
D A T A
Hub and Spokes Coupling
Land AtmLandCoupler Atmosphere DATA DATA Pairwise Coupling
26
Design Strategy:Communication Within ComponentsAll communication in ESMF is handled within components. This means that if an atmosphere is coupled to an ocean, then the Coupler Component is defined on both atmosphere and ocean processors.
ocn_comp atm_comp
processors
atm2ocn _coupler
27
Design Strategy:Uniform Communication API• The same programming interface is used for shared memory, distributed
memory, and combinations thereof. This buffers the user from variations and changes in the underlying platforms.
• The idea is to create interfaces that are performance sensitive to machine architectures without being discouragingly complicated.
• Users can use their own OpenMP and MPI directives together with ESMF communications
ESMF sets up communications in a way that is sensitive to the computing platform and the application structure
28
ESMF Class Structure
DELayoutCommunications
StateData imported or exported
BundleCollection of fields
GridCompLand, ocean, atm, … model
F90
Superstructure
Infrastructure
FieldPhysical field, e.g. pressure
GridLogRect, Unstruct,
etc.
Data Communications
C++
RegridComputes interp weights
CplCompXfers between
GridComps
UtilitiesVirtual Machine, TimeMgr, LogErr, IO, ConfigAttr, Base etc.
ArrayHybrid F90/C++ arrays Route
Stores comm paths
DistGridGrid decomposition
PhysGridMath description
29
ESMF Superstructure Classes• Gridded Component
◦ Models, data assimilation systems - “real code”• Coupler Component
◦ Data transformations and transfers between Gridded Components
• State – Packages of data sent between Components• Application Driver – Generic driver
30
ESMF Infrastructure Data Classes
Model data is contained in a hierarchy of multi-use classes. The user can reference a Fortran array to an Array or Field, or retrieve a Fortran array out of an Array or Field.
• Array – holds a Fortran array (with other info, such as halo size) • Field – holds an Array, an associated Grid, and metadata• Bundle – collection of Fields on the same Grid bundled together
for convenience, data locality, latency reduction during communicationsSupporting these data classes is the Grid class, which represents a numerical grid
31
ESMF Communications• Halo
◦ Updates edge data for consistency between partitions• Redistribution
◦ No interpolation, only changes how the data is decomposed• Regrid
◦ Based on SCRIP package from Los Alamos ◦ Methods include bilinear, conservative
• Bundle, Field, Array-level interfaces
32
ESMF Utilities
• Time Manager• Configuration Attributes (replaces namelists)• Message logging • Communication libraries• Regridding library (parallelized, on-line SCRIP)• IO (barely implemented)• Performance profiling (not implemented yet, may simply use
Tau)
33
Adoption Strategies: Top Down1. Decide how to organize the application as discrete Gridded and Coupler
Components. The developer might need to reorganize code so that individual components are cleanly separated and their interactions consist of a minimal number of data exchanges.
2. Divide the code for each component into initialize, run, and finalize methods. These methods can be multi-phase, e.g., init_1, init_2.
3. Pack any data that will be transferred between components into ESMF Import and Export States in the form of ESMF Bundles, Fields, and Arrays. User data must match its ESMF descriptions exactly.
4. The user must describe the distribution of grids over resources on a parallel computer via the VM and DELayout.
5. Pack time information into ESMF time management data structures.6. Using code templates provided in the ESMF distribution, create ESMF
Gridded and Coupler Components to represent each component in the user code.
7. Write a set services routine that sets ESMF entry points for each user component’s initialize, run, and finalize methods.
8. Run the application using an ESMF Application Driver.
34
Adoption Strategies: Bottom UpAdoption of infrastructure utilities and data structures can follow many different paths. The calendar management utility is a popular place to start, since there is enough functionality in the ESMF time manager to merit the effort required to integrate it into codes and bundle it with an application.
35
OUTLINE
• Overview• ESMF and the Community• Development Status• Design and Principles of ESMF• Resources
36
Documentation• Users Guide
◦ Installation, quick start and demo, architectural overview, glossary• Reference Manual
◦ Overall framework rules and behavior◦ Method interfaces, usage, examples, and restrictions◦ Design and implementation notes
• Developers Guide◦ Documentation and code conventions◦ Definition of compliance
• Requirements Document• Implementation Report
◦ C++/Fortran interoperation strategy• (Draft) Project Plan
◦ Goals, organizational structure, activities
37
User Support
• All requests go through the [email protected] list so that they can be archived and tracked
• Support policy is on the ESMF website• Support archives and bug reports are on the ESMF website -
see http://www.esmf.ucar.edu > Development
Bug reports are under Bugs and support requests are under Lists.
38
Testing and Validation Pages• Accessible from the Development link on the ESMF website• Detailed explanations of system tests• Supported platforms and information about each• Links to regression test archives• Weekly regression test schedule
39
Latest Information
For scheduling and release information, see:
http://www.esmf.ucar.edu > Development
This includes latest releases, known bugs, and supported platforms.
Task lists, bug reports, and support requests are tracked on the ESMF SourceForge site:
http://sourceforge.net/projects/esmf