Petascale Atmospheric Models for the CCSM: New

Developments and Evaluation of Scalable Dynamical Cores

Mark Taylor (Sandia)

Jim Edwards (NCAR/IBM), Christiane Jablownowski (Univ. Michigan), Peter Lauritzen (NCAR), Ram Nair (NCAR), Amik St.Cyr (NCAR)

SciDAC PI Meeting, Seattle, July 2008

Office of Science

U.S. Department of Energy

Why Petascale Climate Models?

• DOE BER Scidac Project to build a first generation Earth System Model

– Based on the Community Climate System Model (CCSM)

– Full carbon cycle (and other cycles) to allow for prescribing greenhouse gas emissions instead of prescribing concentrations

– Improved regional resolution to better assess potential impacts of climate change

– These new capabilities will require petascale performance

• Many other modeling centers have similar goals & plans

Outline

• Largest scalability bottleneck in CCSM: – Dynamical core of atmospheric component

• 2008 NCAR ASP Summer Colloquium on Numerical Techniques for Global Atmospheric Models

– International & collaborative efforts to develop petascale-ready atmospheric dynamical cores

– Focus on model inter-comparison using idealized test cases with “reference” solutions

• Evaluation within the full atmospheric model– No convergence under mesh refinement! Use Williamson

equivalent resolution methodology for Aqua Planet

– Results using 90K processors from a cubed-sphere based dynamical core in the CCSM.

Community Climate System Model(CCSM)

Model Components

• Petascale Ready: scaling to O(100,000) processors at target climate resolutions (10km)

• Land and Ice models are petascale-ready

• Ocean component is near petascale-ready

• Atmosphere component is the performance bottleneck

Community Atmospheric Model(CAM)

• Atmospheric Model: Physics component– Mostly subgrid parametrizations and forcings (convection,

precipitation, radiative forcing, etc.)

– Appear as forcing terms in the equations

– Column based, embarrassingly parallel with 2D domain decomposition

– Responsible for most of the uncertainty

• Atmospheric Model: Dynamics component– Solves the Atmospheric Primitive Equations (compressible Euler

equations in a rotating reference frame with hydrostatic and shallow atmosphere approximation, scale selective dissipation)

– Responsible for most of the variability

– Scalability bottleneck!

The Dynamical Core Scalability Bottleneck

•Most dynamical cores in operational models use latitude-longitude grids: – Well proven. Many good solutions to

the “pole problem”: Spherical harmonics, polar filtering, implicit methods

– But these approaches degrade parallel scalability

– However, scalability continues to improve (see SciDAC BER/ASCR Scientific Application Partnership)

The Dynamical Core Scalability Bottleneck

•Petascale dynamical core:– Quasi-uniform grids avoid the pole problem

– Can use full 2D domain decomposition in horizontal directions

– Each column in the vertical/radial direction kept on processor

– Equations can be solved explicitly, only nearest neighbor communication

– Numerical methods that perform as well as lat/lon based methods remains a challenge.

• NCAR ASP Summer Colloquium on Dynamical Cores– June 2-13, 2008, Boulder CO

– Organizers: Lauritzen, Jablonowski, Taylor, Nair

– NCAR/NSF, NASA and DOE Support

– ~40 graduate students

– 12 keynote lecturers (See upcoming Springer Lecture Notes in Computational Science and Engineering)

– 11 models: GISS BQ, CAM Eulerian, CAM FV isentropic, GEOS FV, GEOS FV-cubed, CSU GCM, GME, HOMME, ICON, OLAM, NCEP GEF, MIT GCM

Summer Colloquium on Dynamical Cores

•Students ran 6 dynamical core test cases (some with several variants)

– Steady-state (3 rotations)– Baroclinic instability (3 rotations)– Pure 3D advection with prescribed winds– Rossby-Haurwitz wave– Mountain-induced Rossby waves– Gravity wave tests (4 variants)

•All models tested with identical initial conditions and run in their operational configurations which included their typical diffusion mechanisms. Most models are hydrostatic and based on the Primitive Equations.

•Results shown here are for medium high horizontal resolution (about 1 degree). Vertical resolution varied from 20 to 60 levels.

•1 TB of data collected, to be put on the Earth System Grid

•https://www.wiki.ucar.edu/display/dycores/Home

Jablonowski and Williamson, A Baroclinic Instability Test Case for Atmospheric Model Dynamical Cores, Q.J.R. Meteorol. Soc. (2006)

– Dynamical core only: no atmospheric physics– L2 error in surface pressure as a function of

time shown below– Converges under mesh refinement to

reference solution (uncertainty in reference solution is yellow shaded region)

Test 2: Baroclinic Instability Test

Test 2: Baroclinic instability. Surface pressure at day 9. The tests starts with balanced initial conditions that are overlaid by a Gaussian hill perturbation. The perturbation grows into a baroclinic wave. Some models show cubed-sphere or icosahedral grid imprinting in the Southern Hemisphere. High order methods show spectral ringing in the 1000mb contour.

GISS-BQ CAM EUL CAM-FV isen

GME

OLAMICON

GEOS-FVcubeGEOS-FV

HOMME

Test 3: Pure Advection. Latitude-height cross section of a 3D slotted ellipse tracer distribution after one revolution around the sphere (day 12). The 3D winds are prescribed. The slotted ellipse has followed a trajectory path with three wave cycles in the vertical direction. The test evaluates the diffusion characteristics of the advection algorithm.

GISS-BQ CAM EUL CAM-FV isen

GME

OLAMICON

GEOS-FVcubeINITIAL STATE

HOMME

Test 4: Westward propigating Rossby-Haurwitz wave. Geopotential height is plotted at 500mb at day 15. The wave number 4 pattern moves westward with minimal change in shape.

GISS-BQ CAM EUL CAM-FV isen

GME

OLAMICON

GEOS-FVcubeGEOS-FV

HOMME

Test 5: Mountain induced Rossby wave. Zonal wind at 700mb, day 15. The test starts with balanced and isothermal initial conditions. A 2-km high Gaussian-hill shaped mountain with half-width 1.5 km is placed at 90°E, 30°N. The mountain triggers Rossby waves. The test evaluates the treatment of the orography and reveals numerical noise (especially at later days).

GISS-BQ CAM EUL CAM-FV isen

GME

OLAMICON

GEOS-FVcubeGEOS-FV

HOMME

Test 6: Pure Gravity Wave Test. Longitude-height cross section along the equator of the potential temperature perturbation at day 4. The Coriolis parameter is set to zero. The test reveals the influence of the diffusion and damping mechanisms in the dynamical cores. Divergence damping (in FV models) leads to a significant decrease in the gravity wave amplitude. Diffusion also influences the sharpness of the gradient.

GISS-BQ CAM_EUL CAM-FV isen

GME

OLAMICON

GEOS-FVcubeGEOS-FV

HOMME

Dynamics and Physics Evaluation

Neale & Hoskins, 2000a: A standard test for AGCMs including their physical parameterizations, Atmos. Sci. Lett.Compare dycore behavior in full atmospheric model (moisture and full physics), but still in an idealized settingStresses dissipation mechanisms and moisture advection much more than pure dynamical tests.

Aqua Planet Experiments

HOMME cubed-sphere dycore integrated in CAM, the Community Atmospheric Model component of the CCSM

CAM Eulerian (global spectral model) and CAM-FV results taken from Williamson, Tellus 2008a, 2008b

Follow Williamson equivalent resolution methodology – No convergence under mesh refinement, as expected due to the nature of many of

the subgrid physics parametrizations

– Strong signal with resolution

– However, agreement between dynamical cores allows us to establish equivalent resolutions

– CAM 3.1 Physics with parameters held fixed for all models and all resolutions

– Compare 1 year means (after suitable spinup)

CAM/HOMME Spectral Element Dynamical Core

h-p Finite element method on a cubed-sphere grid

4th order accurate

1st dynamical core in CAM to conserve both Mass and Energy– Conservation via Compatible Discretization

– Energy conservation even for equations in non-conservation form

– Zero dissipation from the discretization

1st dynamical core in CAM to allow full 2D domain decomposition

Tracer advection: consistent with continuity equation, positive preserving, but not monotone

Resolution Viscosity PRECC PRECL CLDTOT TMQ

EUL T42 5m 1.0E+16 1.71 1.11 0.64 20.21

HOMME 1.9 5m 1.0E+16 1.76 1.14 0.66 20.09

EUL T85 5m 1.0E+15 1.59 1.38 0.60 19.63

HOMME 1.0 5.5m 1.0E+15 1.59 1.43 0.61 19.67

EUL T170 5m 1.5E+14 1.44 1.62 0.55 19.13

HOMME 0.5 5m 1.5E+14 1.48 1.62 0.55 19.36

T340 5m 1.5E+13 1.36 1.75 0.50 18.75

Physics dt

Aqua Planet Global Mean Quantities

Compared to the size of the resolution signal, there is a remarkable agreement between CAM/HOMME and CAM/Eulerian

Precipitation PDFsH

OM

ME

FV

& E

uler

ian

1mm bin-size 10mm bin-size

Aqua Planet Experiment: Zonal DataComparison with FV & Eulerian Dycore

HO

MM

EF

V &

Eul

eria

n

Fixed Mesh Scalability CAM/HOMME

•Good scalability down to 1 element per processor (86,200 processors at 0.25 degree resolution). Higher resolutions will easily scale to even more processors

• Integration rates better than 5 simulated years/day at resolutions down to 0.25 degree

•BGL results: 1 processor per node due to memory constraints. BGP results use 4 processor cores per node. BGP is 4x-8x faster per node.

High Resolution Aqua Planet Experiments

CAM 3.5 Physics

14 month simulations

5min physics timestep

Eulerian T85 physics tunings

Simulations on LLNL BG/L system–0.250 degree: 43200 processors ~1 day per

simulation–0.125 degree: 57600 processors ~3 days per

simulation

High Resolution Results - CAM 3.4 Physics

Summary

• Community is committed to developing petascale-ready earth system models

• Atmospheric component: – Unprecedented collaboration between modeling groups on next

generation dynamical cores initiated at NCAR Colloquim

– CAM/HOMME: Petascale-ready cubed-sphere dynamical core integrated into CAM

– CAM/HOMME: aqua planet test case results agree remarkably well with conventional dynamical cores

• Current Focus: (BER SciDAC projects)– CCSM coupling with land, ocean and ice (Mariana Vertenstein, Tony

Craig, Kate Evans)

– Discontinious Galerken advection schemes for HOMME (Ram Nair, Amik St.Cyr, Henry Tufo)