Computational Design of the CCSM Next Generation Coupler

Tom BettgeTony Craig

Brian Kauffman

National Center for Atmospheric ResearchBoulder, Colorado USA

Draft Presentation: The Fourth International Workshop on Next Generation Climate Models for Advanced High Performance Computing Facilities, March 12-14, 2002, Boulder, Colorado, USA.

Background 1:Current Coupler Shortcomings

• CSM1 load balancing• Parallel Climate Model (PCM) scalability• Flexibility• Extensibility• Portability?

– CSM1 designed primarily for vector architectures

– PCM designed primarily for distributed memory single/multiprocessor clusters

• Code reuse: poor

Background 2:Future Coupler Design

• New CCSM Coupler: Next Generation Coupler (NGC)

• DOE/NSF Proposal– Performance-Portable Extendible Coupler

• scientific requirements• computational requirements

– Formal Software Development Process• isolate requirements (Spring 2000)• state a design (abstractions, code reuse, etc.)• prototyping• code construction• testing and validation

www.ccsm.ucar.edu/models/cpl6

NGC Requirements:Scientific Functionality

• Provide overall control of coupled system• Flexible and extensible: new components, new

exchange information• Flexible coupling frequencies• Process split or time split advancement• Allow computations

– compute interfacial fluxes– mapping between grids– merging of information– temporal accumulation and averaging– diagnostics

• Process data on a variety of grids

NGC Requirements:Computational Functionality

• Portable• Language: F90 or F90 interfaces• Parallelism

– pure shared memory– pure message passing– hybrid

• Flexible coupler/model interface• Sequential or concurrent model execution• Build as single or multiple executables• I/O capability• Error handling

NGC Design:Considerations

• high level command/control• lower level functions --> suggests a layered design strategy

• concurrent or sequential execution• flexibility• generalized error handling --> suggests a library core of functions • generalized grid representations• extensibility --> suggests internal derived data types

Next Generation CouplerFormal Design

MCT*

MPH**

*Model Coupling Toolkit

** Multi-Component Handshaking Library

cplx (cpl6)

MPEU – Message Passing Env Utility

From NASA DAO, provides the following services:• F90 module-style access to MPI• Portable/Flexible definition of types• Support for multiprocessor stdout/stderr• Error handling / shutdown• Timing/Load balance monitoring tools• Sorting Tools• Support for basic derived types on which low-level

classes in the MCT are built

Code used as scaffolding, but utilities might fit well in CCSM.

MCT - Model Coupling Toolkit

Major Attributes:– Maintains model decomposition descriptors (e.g.,

global to local indexing)– Inter- and intra- component communications and

parallel data transfer (routing)– Flexible, extensible, indexible field storage– Time averaging and accumulation– Regridding (via sparse matrix vector multiply)

The MCT eases the construction of coupler computational cores and component-coupler interfaces.

www-unix.mcs.anl.gov/acpi/mct

MPH - Multi-Component Handshaking Library

General Features– built on MPI– establishes MPI communicator for each component– performs component name registration– allows resource allocation for each component– supports different execution modes

www.nersc.gov/research/SCG/acpi/MPH

cplx Architecture

Component Model/Coupler Interface

Use coupler interface modules:

use module ngc_interface_mod.F90 use module ngc_fields_mod.F90

Compile/link shared code (cpl6 modules, MCT, MPH) with model.

Calls:

ngc_init( modelID, localComm )

ngc_sendInit( modelID, flags(:), buffer(:,:) ) ngc_recvInit( modelID, flags(:), buffer(:,:) )

ngc_send( modelID, flags(:), buffer(:,:) ) ngc_recv( modelID, flags(:), buffer(:,:) )

ngc_sendFin( modelID, flags(:), buffer(:,:) )

ngc_finalize( modelID )

cpl6 Implementation

• Initial cpl6 - same interactions as cpl5• Concurrent component execution• Multiple executables• Message passing will be distributed parallel• Matrix multiply will be distributed parallel

cpl6 Coupling Procedure/Issues

• multiple executable, concurrent execution (CCSM2)

Coupled System Processor Layout

PCM

CCSM2

Target?

CSM1

Schedule and FAQ

• cpl6 prototype – exists• cpl6beta – June 2002• cpl6 – Fall 2002

• Will cpl6 be the coupler in the May 2002 CCSM2

release?No. The first release of CCSM2 will contain cpl5.

• Will cpl6 offer new functionality over cpl5?No. For the most part, cpl6 replicates cpl5 functionality. However, the

new coupling framework provides the tools to easily build differing instantiations of a “coupler”. cpl6 is one such instantiation (e.g.., cpl6 is distributed parallel)

cpl [next]

• cpl[next] charter?– user community input to whom? (no coupler WG)– process for introducing/requesting/prioritizing new

features

• features?– “on the fly” SCRIP (multiple options)– CCSM single binary, sequential executable cpl– cpl suitable for cluster of vector machines– concept of one model, multiple grids?