geo387p climate system physics running cesm on stampede · atmosphere atm cam active atmosphere atm...

GEO387P Climate System Physics

Running CESM* on Stampede

05 Feb 2015

* (Community Earth System Model)

What are Climate Models q GCMs are computer-run, mathematical simulators of

the atmosphere, land, ocean and sea ice.

q GCMs express, in mathematical form, the physical processes that dictate the behavior of all components of the climate.

q Climate models are systems of differential equations based on the basic laws of physics, fluid motion, and chemistry. To “run” a model, scientists divide the planet into a 3-dimensional grid, apply the basic equations, and evaluate the results.

q Atmospheric models calculate winds, heat transfer, radiation, relative humidity, and surface hydrology within each grid and evaluate interactions with neighboring points.

What are Climate Models … q GCMs are the principal tool for predicting the

response of the climate to increases in greenhouse gases.

q GCMs include the interaction of the atmosphere with the oceans and with the surface of the Earth, including plants and other ground cover.

q They allow us to test, by mathematical simulation, what should happen to climate, around the world, in response to a wide variety of changes. For example, what climatic effects would follow a major volcanic eruption?

Access to stampede.tacc.utexas.edu q  Apply for TACC accounts at:

https://portal.tacc.utexas.edu/account-request q Once you have account, send me your login name and I will add your name to

a project to use ‘stampede’. q  To log into stampede from use the following command from any x-terminal:

ssh –X [email protected]

Login Shell q  Log default login shell is /bin/bash. You may prefer to change it to /bin/csh,

login to: https://portal.tacc.utexas.edu/, and click on TACC consulting and then click at New for new ticket and ask for change of default shell to "/bin/csh". It will be changed in few hours.

q  To temporary change your shell for current session, type: tcsh or csh or ksh etc. q  To check current shell, type: echo $SHELL q  You may modify a .cshrc file in your home directory. You may copy my .cshrc

from "/home1/01050/shaikh”: At your home directory; type: cp ~shaikh/.cshrc .

Disk Quotas on stampede Three disk spaces are on stampede.

q  $Home directory has 5GB quota limit (command: lfs quota $HOME). It is small sized but backed up daily.

q  $WORK has a quota limit of 1000GB (command: lfs quota $WORK). Files are never deleted within quota but no backup. System alias cdw to change directory to $WORK.

q  $SCRATCH has no quota limit. Files accessible time is 10 days but in reality files are there after for months. System alias "cds" to change directory to $SCRATCH

q  $ARCHIVE for long-term storage. Login to archiver using command: ssh $ARCHIVER. Command: scp –r dirname ${ARCHIVER} to send data to archiver.

q  Keep model scripts & code changes in your $HOME directory, CESM model & input data on $WORK directory and model simulations on $SCRATCH disk.

Downloading NCAR’s CESM Model q CESM stands for “Community Earth System Model”. q Download CESM in your work directory. To change directory to $WORK, type: cdw q  To check model versions available: svn list --username guestuser --password friendly https://svn-ccsm-models.cgd.ucar.edu/cesm1/release_tags q To download CESM1_2_1 svn co --username guestuser --password friendly https://svn-ccsm-models.cgd.ucar.edu/cesm1/release_tags/cesm1_2_1 cesm1_2_1 q  Also create a directory named “inputdata” in $WORK directory, to store model input

data files; downloaded automatically from NCAR. cdw mkdir inputdata

Model and Machine Model Settings Before compiling the model, we need settings for: q Machine/Hardware: Local modules, libraries, compiler information, input data

directory and model simulation location etc., q Model Component Set: What active, data or stub model components (atm, lnd,

ocn, ice, glc, cpl etc,) to be included in the simulation. We may run active ocean model pop2, or data ocean model slab-ocean-model or prescribed sst to save computational time. For each component, we may choose, active, data or stub model.

q  Component Resolutions: Each model component can run at different resolution independent of other components. For example land, atmosphere, ocean, can run at different resolutions.

Local Settings q  Make following changes in directory: ${WORK}/cesm1_2_1/scripts/ccsm_utils/

Machines!q  Create a new file: env_mach_specific.utexas with following content. !

!setenv FC /opt/apps/intel13/mvapich2/1.9/bin/mpif90!setenv CC /opt/apps/intel13/mvapich2/1.9/bin/mpicc!setenv MPI_PATH /opt/apps/intel13/mvapich2/1.9/lib!setenv MPI_LIB_NAME mpich!setenv NETCDF_PATH /opt/apps/intel13/netcdf/3.6.3/x86_64!setenv LAPACK_LIBDIR /home1/01050/shaikh/LIB/lapack-3.4.2!

!q  Added these lines at the bottom of file: config_machines.xml! <machine MACH="utexas">! <DESC>TACC intel</DESC> ! <OS>LINUX</OS> ! <COMPILERS>intel</COMPILERS> ! <EXEROOT>$ENV{SCRATCH}/$CASE/build</EXEROOT>! <RUNDIR>/$ENV{SCRATCH}/$CASE/run</RUNDIR>! <DIN_LOC_ROOT>$ENV{WORK}/inputdata</DIN_LOC_ROOT>! <GMAKE_J>12</GMAKE_J>! <MAX_TASKS_PER_NODE>32</MAX_TASKS_PER_NODE>! <PES_PER_NODE>16</PES_PER_NODE>! </machine>!q  Copy mkbatch.stampede to mkbatch.utexas!

q  You may copy these files from: /work/01050/shaikh/cesm1_2_1/scripts/ccsm_utils/Machines !

CESM Components q  CESM consists of five geophysical models: atmosphere (atm), sea-ice (ice),

land (lnd), ocean (ocn), and land-ice or glacier (glc), plus a coupler (cpl) that coordinates the models and passes information between them.

q  Each model may have "active," "data," "dead," or "stub" component version allowing for a variety of "plug and play" combinations.

q  During the course of a CESM run, the model components integrate forward in time, periodically stopping to exchange information with the coupler. The coupler meanwhile receives fields from the component models, computes, maps, and merges this information, then sends the fields back to the component models. The coupler brokers this sequence of communication interchanges and manages the overall time progression of the coupled system.

q Model components are written primarily in Fortran 90.

CESM Components q  The active (dynamical) components are generally fully prognostic, and they are

state-of-the-art climate prediction and analysis tools.

q  Because the active models are relatively expensive to run, data models that cycle input data are included for testing, spin-up, and model parameterization development.

q  The dead components generate scientifically invalid data and exist only to support technical system testing. The dead components must all be run together and should never be combined with any active or data versions of models.

q  stub components exist only to satisfy interface requirements when the component is not needed for the model configuration (e.g., the active land component forced with atmospheric data does not need ice or ocn components, so ice, ocn, and glc stubs are used).

Summary of CESM Components

Model Type

Model Name

Component Name Type

atmosphere atm cam active

atmosphere atm datm data

atmosphere atm xatm dead

atmosphere atm satm stub

land lnd clm active

land lnd dlnd data

land lnd xlnd dead

land lnd slnd stub

river-runoff rof rtm active

river-runoff rof drof data

river-runoff rof xrof dead

river-runoff rof srof stub

Model Type

Model Name

Component Name Type

ocean ocn pop active

ocean ocn docn data

ocean ocn xocn dead

ocean ocn socn stub

seaice ice cice active

seaice ice dice data

seaice ice xice dead

seaice ice sice stub

land-ice glc cism active

land-ice glc sglc stub

ocean-wave wav xwav dead

ocean-wave wav swav stub

coupler cpl cpl active

CESM Components Set Categories

Components Description

A datm,dlnd,dice,docn,sglc All DATA components with stub glc (used primarily for testing)

B cam,clm,cice,pop2,sglc FULLY ACTIVE components with stub glc

C datm,dlnd,dice,pop2,sglc POP active with data atm, lnd(runoff), and ice plus stub glc

D datm,slnd,cice,docn,sglc CICE active with data atm and ocean plus stub land and glc

E cam,clm,cice,docn,sglc CAM, CLM, and CICE active with data ocean (som mode) plus stub glc

F cam,clm,cice,docn,sglc CAM, CLM, and CICE(prescribed mode) active with data ocean (sstdata mode) plus stub glc

G datm,dlnd,cice,pop2,sglc POP and CICE active with data atm and lnd(runoff) plus stub glc

H datm,slnd,cice,pop2,sglc POP and CICE active with data atm and stub land and glc

I datm,clm,sice,socn,sglc CLM active with data atm and stub ice, ocean, and glc

S satm,slnd,sice,socn,sglc All STUB components (used for testing only)

X xatm,xlnd,xice,xocn,sglc All DEAD components except for stub glc (used for testing only)

Components Set Naming TIME_ATM[%phys]_LND[%phys]_ICE[%phys]_OCN[%phys]_ROF[%phys]_GLC[%phys]_WAV[%phys]!TIME = model time period (e.g. 2000, 20TR, RCP8...) !ATM = [CAM4, CAM5, DATM, SATM, XATM]! !LND = [CLM40, CLM45, DLND, SLND, XLND] ! !ICE = [CICE, DICE, SICE, SICE] ! ! !OCN = [POP2, DOCN, SOCN, XOCN, AQUAP] ! !ROF = [RTM, DROF, SROF, XROF] ! ! !GLC = [CISM1, SGLC, XGLC] ! ! ! !WAV = [SWAV, XWAV] ! ! ! ! !!v  The OPTIONAL %phys attributes specify sub-modes of the given system. For example

DOCN%DOM is the DOCN data ocean (rather than slab-ocean) mode. v  Each model compset can be specified by its alias, short name and long name. Example of

equivalent alias, short name and long name - alias: BC5CN - short name: B_2000_CAM5_CN - long name: 2000_CAM5_CLM40%CN_CICE_POP2_RTM_SGLC_SWAV - alias: FAMIPC5CN - short name: F_AMIP_CAM5_CN - long name: AMIP_CAM5_CLM40%CN_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV

Components Sets (Time Period) q  Time Period (first four characters)!1850 => pre-industrial !2000 => present day !20TR => transient 1850 to 2000 !5505 => transient 1955 to 2005 !9205 => transient 1992 to 2005 !RCP8 => transient RCP8.5 future scenario!RCP6 => transient RCP6.0 future scenario!RCP4 => transient RCP4.5 future scenario!RCP2 => transient RCP2.6 future scenario!NUKE => Nuclear winter hypothetical scenario (based on RCP4.5)!1996 => present day with conditions for solar minimum in 1996!AMIP => transient for "stand-alone" CAM (1979 startdate)!GEOS => GEOS5 metereology for "stand-alone" CAM !!q  CAM !CAM4% => cam4 physics !CAM5% => cam5 physics !CAM[45]%WCCM => CAM WACCM with daily solar data and SPEs: !CAM[45]%WCMX => CAM WACCM-X: !CAM[45]%WCSC => CAM WACCM specified chemistry: !CAM[45]%WCBC => CAM WACCM with the stratospheric black carbon CARMA model: !CAM[45]%WCSF => CAM WACCM with sulfur chemistry and the sulfate CARMA model: !CAM[45]%FCHM => CAM super_fast_llnl chemistry: !CAM[45]%TMOZ => CAM trop_mozart chemistry: !CAM[45]%MOZM => CAM trop_mozart_mam3 chemistry: !CAM[45]%MOZS => CAM trop_mozart_soa chemistry: !CAM[45]%SMA3 => CAM trop_strat_mam3 chemistry: !CAM[45]%SMA7 => CAM trop_strat_mam7 chemistry: !CAM[45]%SSOA => CAM trop_strat_soa chemistry: !CAM[45]%RCO2 => CAM CO2 ramp: !

Components Sets q CLM !note: [^_]* means match zero or more of any character BUT an underbar.!(in other words make sure there is NOT a underbar before the string afterwards)!!CLM40 => clm4.0 Physics!CLM40%[^_]*SP => clm4.0 Satellite phenology!CLM40%[^_]*CN => clm4.0 Carbon Nitrogen!CLM40%[^_]*CNDV => clm4.0 Carbon Nitrogen Dynamic Vegetation!CLM40%[^_]*CROP => clm4.0 Prognostic crop!CLM40%[^_]*SNCR => clm4.0 SNICAR radiative forcing calculation on!!CLM45 => clm4.5 Physics!CLM45%[^_]*SP => clm4.5 Satellite phenology!CLM45%[^_]*CN => clm4.5 Carbon Nitrogen Biogeochemistry (BGC) (as in CLM4.0)!CLM45%[^_]*CNDV => clm4.5 Carbon Nitrogen BGC with Dynamic Vegetation!CLM45%[^_]*BGC => clm4.5 BGC (CN with vertically resolved soil BGC)!CLM45%[^_]*CROP => clm4.5 Prognostic crop!CLM45%[^_]*VIC => clm4.5 VIC hydrology!CLM40%[^_]*SNCR => clm4.0 SNICAR radiative forcing calculation on!CLM45%[^_]*BGCDV => clm4.5 BGC (CN with vertically resolved soil BGC) with dynamic veg! !q CICE !CICE => prognostic cice!CICE%PRES => prescribed cice !!q POP2 !POP2 => POP2 default!POP2%ECO => POP2/Ecosystem!POP2%DAR => Darwin marine ecosystem (not supported in community releases) !!q RTM !RTM => default RTM model!RTM%FLOOD => RTM model with flood !

Components Sets q DATM !DATM%QIA => QIAN atm input data (1948-1972)!DATM%CRU => CRUNCEP atm input data for (1901-2010)!DATM%S1850 => CPL history atm input data!DATM%1PT => single point tower site atm input data!DATM%NYF => COREv2 datm normal year forcing!DATM%IAF => COREv2 datm interannual year forcing! !q DLND !DLND%NULL => dlnd_mode is NULL , dlnd_sno_mode is NULL!DLND%SCPL => dlnd_mode is NULL , dlnd_sno_mode is CPLHIST (used for TG)!DLND%LCPL => dlnd_mode is CPLHIST, dlnd_sno_mode is NULL! !q DROF !DROF%NYF => COREv2 drof normal year forcing!DROF%IAF => COREv2 drof interannual year forcing!DROF%NULL => null mode! !q DICE !DICE%SSMI => dice mode is ssmi ! !DICE%SIAF => dice mode is ssmi_iaf !DICE%PRES => dice mode is prescribed!DICE%COPY => dice mode is copy!DICE%NULL => dice mode is null !!q DOCN !DOCN%NULL => docn null mode!DOCN%SOM => docn slab ocean mode!DOCN%DOM => docn data mode!DOCN%US20 => docn us20 mode!DOCN%COPY => docn copy mode !!

alias: A longname: 2000_DATM%NYF_DLND%NULL_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV !alias: B longname: 2000_CAM4_CLM40%SP_CICE_POP2_RTM_SGLC_SWAV !alias: BCN longname: 2000_CAM4_CLM40%CN_CICE_POP2_RTM_SGLC_SWAV !alias: BC5 longname: 2000_CAM5_CLM40%SP_CICE_POP2_RTM_SGLC_SWAV !alias: BC5CN longname: 2000_CAM5_CLM40%CN_CICE_POP2_RTM_SGLC_SWAV !alias: B2000CNCHM longname: 2000_CAM4%FCHM_CLM40%CN_CICE_POP2_RTM_SGLC_SWAV !alias: B1850 longname: 1850_CAM4_CLM40%SP_CICE_POP2_RTM_SGLC_SWAV !alias: B1850C5 longname: 1850_CAM5_CLM40%SP_CICE_POP2_RTM_SGLC_SWAV !alias: B1850CN longname: 1850_CAM4_CLM40%CN_CICE_POP2_RTM_SGLC_SWAV !alias: B1850C5CN longname: 1850_CAM5_CLM40%CN_CICE_POP2_RTM_SGLC_SWAV !alias: B1850W longname: 1850_CAM4%WCCM_CLM40%SP_CICE_POP2_RTM_SGLC_SWAV !alias: F1850 longname: 1850_CAM4_CLM40%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV !alias: F1850C5 longname: 1850_CAM5_CLM40%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV !alias: F longname: 2000_CAM4_CLM40%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV !alias: FC5 longname: 2000_CAM5_CLM40%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV !alias: FAMIP longname: AMIP_CAM4_CLM40%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV !alias: FAMIPC5 longname: AMIP_CAM5_CLM40%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV !alias: FCN longname: 2000_CAM4_CLM40%CN_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV !alias: FAMIPCN longname: AMIP_CAM4_CLM40%CN_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV !alias: FAMIPC5CN longname: AMIP_CAM5_CLM40%CN_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV!alias: I1PT longname: 2000_DATM%1PT_CLM40%SP_SICE_SOCN_RTM_SGLC_SWAV !alias: I1PTCLM45 longname: 2000_DATM%1PT_CLM45%SP_SICE_SOCN_RTM_SGLC_SWAV !alias: I20TRCLM45 longname: 20TR_DATM%QIA_CLM45%SP_SICE_SOCN_RTM_SGLC_SWAV !alias: ICN longname: 2000_DATM%QIA_CLM40%CN_SICE_SOCN_RTM_SGLC_SWAV !alias: ICLM45CN longname: 2000_DATM%QIA_CLM45%CN_SICE_SOCN_RTM_SGLC_SWAV !alias: I1850CN longname: 1850_DATM%QIA_CLM40%CN_SICE_SOCN_RTM_SGLC_SWAV !alias: I1850CLM45CN longname: 1850_DATM%QIA_CLM45%CN_SICE_SOCN_RTM_SGLC_SWAV !alias: I1850CLM45CNF longname: 1850_DATM%QIA_CLM45%CN_SICE_SOCN_RTM%FLOOD_SGLC_SWAV ! !q  For the complete list of compsets:! cd $WORK/cesm1_2_1/scripts! create_newcase –l compsets !

Sample of Model Components Set

Grid Naming q  The notation is

!a%name_l%name_oi%name_r%name_m%mask_g%name_w%name!

q  The model component resolution can be specified by its alias, short name or long name. Example of equivalent alias, short name and long name:

-   alias: f19_g16 (atm/lnd_ocn/ice)

-   short name: 1.9x2.5_gx1v6

-   long name = a%1.9x2.5_l%1.9x2.5_oi%gx1v6_r%r05_m%gx1v6_g%null_w%null

atm lnd ocn-ice river land mask glacier wave

q  "[dlat]x[dlon]" are regular lon/lat finite volume grids where dlat and dlon are the approximate grid spacing. The shorthand convention is "fnn" where nn is generally a pair of numbers indicating the resolution. An example is 1.9x2.5 or f19 for the approximately "2-degree" finite volume grid. Note that CAM uses an [nlat]x[nlon] naming convection internally for this grid.

q  "Tnn" are spectral lon/lat grids where nn is the spectral truncation value for the resolution. The shorthand name is identical. An example is T85.

q  "ne[X]np[Y]" are cubed sphere resolutions where X and Y are integers. The short name is generally ne[X]. An example is ne30np4 or ne30.

q  "pt1" is a single grid point.

q  "gx[D]v[n]" is a displaced pole grid where D is the approximate resolution in degrees and n is the grid version. The short name is generally g[D][n]. An example is gx1v6 or g16 for a grid of approximately 1-degree resolution.

q  "tx[D]v[n]" is a tripole grid where D is the approximate resolution in degrees and n is the grid version.

Sample of Model Grid model grid: a%1x1_vancouverCAN_l%1x1_vancouverCAN_oi%1x1_vancouverCAN_r%null_m%reg_g%null_w%null !model grid: a%5x5_amazon_l%5x5_amazon_oi%5x5_amazon_r%null_m%reg_g%null_w%null (alias 5amazon)!model grid: a%360x720cru_l%360x720cru_oi%360x720cru_r%r05_m%360x720cru_g%null_w%null !model grid: a%T31_l%T31_oi%gx3v7_r%r05_m%gx3v7_g%null_w%null (alias T31_g37)!model grid: a%T85_l%T85_oi%gx1v6_r%r05_m%gx1v6_g%null_w%null (alias T85_g16)!model grid: a%T85_l%0.9x1.25_oi%gx1v6_r%r05_m%gx1v6_g%null_w%null (alias T85_f09_g16)!model grid: a%T85_l%0.9x1.25_oi%tx0.1v2_r%r05_m%gx1v6 _g%null_w%null (alias T85_f09_t12)!model grid: a%T85_l%0.23x0.31_oi%tx0.1v2_r%r05_m%gx1v6_g%null_w%null (alias T341_f02_t12)!model grid: a%0.23x0.31_l%0.23x0.31_oi%gx1v6_r%r05_m%gx1v6_g%null_w%null (alias f02_g16)!model grid: a%0.47x0.63_l%0.47x0.63_oi%gx1v6_r%r05_m%gx1v6_g%null_w%null (alias f05_g16)!model grid: a%0.9x1.25_l%0.9x1.25_oi%gx1v6_r%r05_m%gx1v6_g%null_w%null (alias f09_g16)!model grid: a%1.9x2.5_l%1.9x2.5_oi%gx1v6_r%r05_m%gx1v6_g%null_w%null (alias f19_g16)!model grid: a%0.23x0.31_l%0.23x0.31_oi%0.23x0.31_r%r05_m%gx1v6_g%null_w%null (alias f02_f02)!model grid: a%0.47x0.63_l%0.47x0.63_oi%0.47x0.63_r%r05_m%gx1v6_g%null_w%null (alias f05_f05)!model grid: a%0.9x1.25_l%0.9x1.25_oi%0.9x1.25_r%r05_m%gx1v6_g%null_w%null (alias f09_f09)!model grid: a%1.9x2.5_l%1.9x2.5_oi%1.9x2.5_r%r05_m%gx1v6_g%null_w%null (alias f19_f19)!model grid: a%2.5x3.33_l%2.5x3.33_oi%2.5x3.33_r%r05_m%gx1v6_g%null_w%null (alias f25_f25)!model grid: a%ne16np4_l%ne16np4_oi%gx3v7_r%r05_m%gx3v7_g%null_w%null (alias ne16_g37)!model grid: a%ne240np4_l%ne240np4_oi%ne240np4_r%r01_m%gx1v6_g%null_w%null (alias ne240_ne240)!model grid: a%1.9x2.5_l%1.9x2.5_oi%1.9x2.5_r%r05_m%gx1v6_g%gland10_w%null (alias f19_f19_gl10)!model grid: a%0.9x1.25_l%0.9x1.25_oi%0.9x1.25_r%r05_m%gx1v6_g%gland10_w%null (alias f09_f09_gl10)!model grid: a%T31_l%T31_oi%gx3v7_r%r05_m%gx3v7_g%gland10_w%null (alias T31_g37_gl10)!model grid: a%1.9x2.5_l%1.9x2.5_oi%hc2b7_r%r05_m%gx1v6_g%gland10_w%null (alias f19_g16_gl10)!model grid: a%0.9x1.25_l%0.9x1.25_oi%gx1v6_r%r05_m%gx1v6_g%gland10_w%null (alias f09_g16_gl10)!model grid: a%T31_l%T31_oi%T31_r%r05_m%gx3v7_g%gland5UM_w%null (alias T31_T31)!model grid: a%1.9x2.5_l%1.9x2.5_oi%1.9x2.5_r%r05_m%gx1v6_g%gland5UM_w%null (alias f19_f19)!model grid: a%0.9x1.25_l%0.9x1.25_oi%0.9x1.25_r%r05_m%gx1v6_g%gland5UM_w%null (alias f09_f09)!model grid: a%1.9x2.5_l%1.9x2.5_oi%gx1v6_r%r05_m%gx1v6_g%gland5UM_w%null (alias f19_g16)! q For the complete list of grids:! cd $WORK/cesm1_2_1/scripts! create_newcase –l grids !

CESM 3-step Compilation Script #!/bin/ksh # location: /home1/01050/shaikh/CSM4/test.bld #================================================================== # step1: create new case #================================================================== export CASE=test export COMPSET=FAMIPC5 export RES=f09_g16 export script_dir=${WORK}/cesm1_2_1/scripts export OS=utexas cd ${WORK}; rm -rf ${CASE} cd ${script_dir}; rm -rf ${CASE} ${script_dir}/create_newcase \ -case ${CASE} \ -compset ${COMPSET} \ -compiler intel \ -res ${RES} \ -mach ${OS} #====================================================== # step2: User Changes (xml/F90 files) #====================================================== #rm -rf $script_dir/${CASE}/SourceMods/src.clm/*.F90 cd ${script_dir}/${CASE} #cp ${model_changes}/*.F90 $script_dir/${CASE}/SourceMods/src.clm #====================================================== # Change xml values #====================================================== cesm_setup –clean xmlchange STOP_OPTION=nmonths xmlchange STOP_N=1200 xmlchange REST_OPTION=nmonths xmlchange REST_N=12 #xmlchange CLM_CONFIG_OPTS="-phys clm4_5 -bgc cn" xmlchange GMAKE_J=12 cesm_setup #====================================================== # step3: build CASE #====================================================== ${CASE}.build

CESM compile script

q  The first line defines Unix shell for the script. Examples: /bin/ksh, /bin/bash, /bin/csh,

/bin/tcsh. The script shell may be different from your login shell.

q  Except for the first line with #!, all lines starting with # are comments.

#!/bin/ksh # location: /home1/01050/shaikh/CSM4/test.bld

export CASE=test export COMPSET=FAMIPC5 export RES=f09_g16 export script_dir=${WORK}/cesm1_2_1/scripts export OS=utexas

q  Define environment variables. For ksh,bash & sh shells, its format is:

export variable=value

q  For csh & tcsh, environment variables are defined by:

setenv variable value

q  To use environment variables, add $ sign before the variable: $variable or ${variable}. Example: ${CASE}, ${COMPOSET} ${script_directory}, ${OS}, ${RES}.

q  Environment variables are case sensitive.

q  CASE is case name for the run.

q  COMPSET is model components set name: FAMIPC5 is for F category of compset (atm, lnd active model, ocn & seaice data model, with AMIP SST. C5 refers to CAM5 physics.

CESM compilation script q  RES is model resolution: f09_g16 is 0.9ox1.25o for lnd & atm and 1o for ocn and seaice..

q  script_dir is directory having create_newcase command.

q  OS is utexas for variables setting for local site.

q  COMPILER is intel for Fortran, C, MPI etc. settings.

#====================================================== # step2: User Changes (xml/F90 files) #====================================================== #rm -rf $script_dir/${CASE}/SourceMods/src.clm/*.F90 cd ${script_dir}/${CASE} #cp ${model_changes}/*.F90 $script_dir/${CASE}/SourceMods/src.clm

q  If you have modified any of land model files or created some new ones, these be copied to $script_dir/${CASE}/SourceMods/src.clm. Similarly, atm model files be copied to $script_dir/${CASE}/SourceMods/src.cam. Directory $script_dir/${CASE}/SourceMods has 9 subdirectories (src.clm, src.cam, src.pop2, src.cice, src.rtm, src.drv, src.sglc, src.share, src.swav) where user’s modified or new files can be copied. User files will override any model files with same name.

q  This step is option. It only needed if you have files to be included in the model. Uncomment lines with rm and cp. ${model_changes}is name of the directory containing user files.

CESM Compilation Script #=========================================================== # Change environment variables saved in xml file (optional) #=========================================================== cesm_setup –clean xmlchange STOP_OPTION=nmonths xmlchange STOP_N=1200 xmlchange REST_OPTION=nmonths xmlchange REST_N=12 #xmlchange CLM_CONFIG_OPTS="-phys clm4_5 -bgc cn" xmlchange GMAKE_J=12 cesm_setup

q  In first step, create_newase command creates five xml files that contain environment variables for this case; located in directory: ${script_dir}/${CASE}.The files are:

q  env_case.xml (27 variables for this case), Sets case specific variables (e.g. model components, model and case root directories). These variables cannot be modified after a case has been created. To make changes, re-run create_newcase with different options.

q  env_mach_pes.xml (54 variables), Sets component machine-specific processor layout. The settings in this are critical to a well-load-balanced simulation. These defines how many processors be allocated to each model component.

q  env_build.xml (71 variables), Sets model build settings, including component resolutions and component configuration options.

q  env_run.xml (168 variables) Sets run-time settings such as length of run, frequency of restarts, output of coupler diagnostics, and short-term and long-term archiving. See run initialization variables , run stop variables , run restart control variables etc.

q  Values in xml files may be changed using xmlchange command as shown above.

Sample of xml File Contents #====================================================== # partial contents of env_run.xml #======================================================  <entry id="RUNDIR" value="//scratch/01050/shaikh/$CASE/run" />  <entry id="CCSM_REPOTAG" value="cesm1_2_1" />  <entry id="RUN_TYPE" value="startup" />  <entry id="RUN_STARTDATE" value="0001-01-01" />  <entry id="RUN_REFDATE" value="0001-01-01" />  <entry id="STOP_OPTION" value="nmonths" />  <entry id="STOP_N" value="1200" />  <entry id="CCSM_CO2_PPMV" value="367.0" />  <entry id="ATM_DOMAIN_FILE" value="domain.lnd.fv0.9x1.25_gx1v6.090309.nc" />

CESM Compilition Script # step3: build CASE ${CASE}.build

q  The step compile all model components in directory: ${EXEROOT} set to $SCRATCH/$CASE/build. An executable file cesm.exe is created in build directory.

q  In ${RUNDIR}, set to $SCRATCH/$CASE/run, namelists are created for each model component. Namelist contents can be freely changed for model simulation. User can change any of model files or parameters defined in namelist. Sample:

&cam_inparm bnd_topo = '/work/01050/shaikh/inputdata/atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027.nc' dtime = 1800 ncdata = '/work/01050/shaikh/inputdata/atm/cam/inic/fv/cami-mam3_0000-01-01_0.9x1.25_L30_c100618.nc' print_energy_errors = .false. / &clm_inparm albice = 0.60,0.40 co2_type = 'diagnostic' create_crop_landunit = .false. dtime = 1800 fatmlndfrc = '/work/01050/shaikh/inputdata/share/domains/domain.lnd.fv0.9x1.25_gx1v6.090309.nc' finidat = ' ' fpftcon = '/work/01050/shaikh/inputdata/lnd/clm2/pftdata/pft-physiology.c130503.nc’ fsurdat = '/work/01050/shaikh/inputdata/lnd/clm2/surfdata_map/surfdata_0.9x1.25_simyr2000_c130418.nc' / &chem_surfvals_nl ch4vmr = 1760.0e-9 co2vmr = 367.0e-6 f11vmr = 653.45e-12 f12vmr = 535.0e-12 n2ovmr = 316.0e-9 /

CESM Run Script #!/bin/csh -f # job submission: sbatch test.run # view job status: showq -u (will show job ID) # cancel job: scancel job_ID #============================================================================= #SBATCH -J test # job name #SBATCH -p normal # queue priority: normal, development #SBATCH -t 48:00:00 # run time (hh:mm:ss) #SBATCH -n 192 # total number of mpi tasks requestions ##SBATCH -A CarbonCycling # charge run to project account #============================================================================= setenv CASE test setenv N ${SLURM_NTASKS} setenv RUNDIR ${SCRATCH}/${CASE}/run setenv BLDDIR ${SCRATCH}/${CASE}/build cd ${RUNDIR} setenv LID "`date +%y%m%d-%H%M%S`" # ------------------------------------------------------------------------- # Create and cleanup the timing directories # ------------------------------------------------------------------------- if !(-d $RUNDIR/timing) mkdir $RUNDIR/timing if !(-d $RUNDIR/timing/checkpoints) mkdir $RUNDIR/timing/checkpoints rm -f $RUNDIR/timing/ccsm_timing* # ------------------------------------------------------------------------- # Run model # ------------------------------------------------------------------------- sleep 25 cd $RUNDIR echo "`date` -- CSM EXECUTION BEGINS HERE" ibrun -np ${N} ${BLDDIR}/cesm.exe >&! cesm.log.$LID wait echo "`date` -- CSM EXECUTION HAS FINISHED"

CESM Run Script #!/bin/csh -f # location: /home1/01050/shaikh/CAM4/test.run # job submission: sbatch test.run # view job status: showq -u (will show job ID) # cancel job: scancel job_ID #============================================================================= #SBATCH -J test #SBATCH -p normal #SBATCH -t 48:00:00 #SBATCH -n 192 ##SBATCH -A CarbonCycling

q  Command to submit job on stampede is: sbatch script_name. If successful, it will display a job_id.

q  To check status of job: showq –u

q  To cancel running/waiting job: scancel job_id

q  -A account Charge resource used by this job to specified account. You need to specify only if you have more than one project to charge cpu-hours.

q  -J jobname This is the name appearing in job queue.

q  -p job_priority normal or development.

q  -n number_of_tasks Maximum 4096 for normal and 256 for development queue.

q  -t time Set limit on total run time, in clock hours, for the job. 48 hours for normal and 2 hours for development job

setenv CASE test setenv N ${SLURM_NTASKS} setenv RUNDIR ${SCRATCH}/${CASE}/run setenv BLDDIR ${SCRATCH}/${CASE}/build cd ${RUNDIR} setenv LID "`date +%y%m%d-%H%M%S`"

q  Set environment variables.

q  Time stamp is created in form: yymmdd-HHMMSS example: 150205-090555

CESM Run Script # ------------------------------------------------------------------------- # Run model # ------------------------------------------------------------------------- sleep 25 cd $RUNDIR echo "`date` -- CSM EXECUTION BEGINS HERE" ibrun -np ${N} ${BLDDIR}/cesm.exe >&! cesm.log.$LID wait echo "`date` -- CSM EXECUTION HAS FINISHED"

q  sleep 25 cause process to wait 25 second.

q  echo command write arguments on screen.

q  ibrun to launch mpi application, -np number_of_cores.

q  wait wait till all processor/cores have finished their jobs.

q  date command displays current date & time:

Thu Feb 05 09:05:55 CST 2015

#------------------------------------------------------------------------- # batch environment variables #------------------------------------------------------------------------- SLURM_JOB_ID batch job id assigned by SLURM upon submission SLURM_JOB_NAME user-assigned job name SLURM_NNODES number of nodes SLURM_NTASKS total number of tasks SLURM_QUEUE queue (partition) SLURM_SUMIT_DIR directory of submission SLURM_TASKS_PER_NODE number of tasks per node SLURM_TACC_ACCOUNT TACC account charged

Cost of Running CESM on Stampede q Computational cost depends on the model components and their

resolutions.

q  For a fully coupled model, e.g. BC5CN (all active component, with CAM5 physics and CN model) at 1o resolution ( land & atmosphere at 0.9oX1.25o, and ocean & sea-ice at 1o resolution) will need 1200 cpu-hrs/model-year (or simply, 100 cpu-hrs/model_month).

q On the other side, simulation with active land only e.g. ICLM45CN (land model, with CLM45 and CN) at same resolution, will need 3.2 cpu-hrs/model-year.

q  Stampede has 900 millions cpu-hrs annually and around 10% of its computational time is reserved for UT faculty/researchers at no cost. For the past five years we have been provided 2-3 millions cpu-hrs annually.

q Our current allocation expires on March 31 and 423,000 cpu-hrs are still available as of today.

geo387p climate system physics running cesm on stampede · atmosphere atm cam active atmosphere atm...

Documents