Blue Template for Slide Presentations

Grid refinement in ROMS

1000 m

5000 m200 m1Two-way grid refinementSST for:USeast grid = 5 kmCarolinas grid = 1 kmGrid refinement of the ocean and wave models is required to allow increased resolution in coastal areas.

Parent grid

0L0Mjust look at rho points for now1Lm1MmChild grid connectivity (for Nref=5)Parent grid rho points4 rho points to left and bottom3 rho points to right and top

Child grid rho pointsParent - child interfaceLower left and Upper right parent psi pointsto identify child region(create_nested_grid.m)Inner child region 0L0M-3-3M+2L+2LmMm11Time steppingmain3d.F (dtp and dtc)# ifdef REFINED_GRID! get data from parent grid at time of parent. Only need! to do this once per child loop. IF (ng.gt.1) THEN IF (get_refdata(ng).eq.TRUE.) THEN CALL get_2dparent_data (ng, TILE) CALL get_3dparent_data (ng, TILE) END IF END IF! Put the child data back into the parent grid. IF ((ng.lt.Ngrids).and.(iic(ng).gt.1)) THEN CALL step3dref_t (ng, TILE) CALL set_2dchild_data (ng, TILE) CALL set_depth (ng, TILE) CALL set_3dchild_data (ng, TILE) END IF# endif# ifdef REFINED_GRID! interpolate the parent data to child time. IF (ng.gt.1) THEN CALL set_2dparent_data (ng, TILE) CALL set_3dparent_data (ng, TILE) END IF# endif

mian model time step computations ..

..Parent (grid 1)Child (grid 2)126211555545555545555545dtpdtc! Read in initial conditions from initial NetCDF file.! CALL get_state (ng, iNLM, 1, INIname(ng), IniRec, Tindex)

#ifdef REFINED_GRID!-----------------------------------------------------------------------! Compute indices of children grid locations in the parent grid.! For 2-way, this needs to be done for all grids, not just ng>1.!----------------------------------------------------------------------- IF (ng.lt.NestedGrids) THEN CALL init_child_hindices (ng, TILE) END IF IF (ng.gt.1) THEN CALL init_parent_hindices (ng, TILE) END IF!-----------------------------------------------------------------------! Obtain initial boudnary conditions from the parent data.!----------------------------------------------------------------------- IF (ng.gt.1) THEN CALL get_2dparent_data (ng, TILE)# ifdef SOLVE3D CALL get_3dparent_data (ng, TILE)# endif END IF#endifinitial.F145iic=03iic=13636iic=236How to prepare a refined grid application for ROMS1) Create child grid 2) interp better bathy to child grid3) Match parent - child bathy + mask4) 3D init and climatology5) Surface forcings 6) roms input file 7) coawst.bash 8) run it

1) create child gridTools/mfiles/mtools/create_nested_grid.m5) set create_child_grid=1this calls parentchild_grid.m

1) enter parent (coarse) grid file2) enter child (fine) grid file3) Istr, Jstr, Iend, Jend4) scale (3 or 5)2) interp better bathy to child gridThe bathy in the grid you just created is from the parent. You need to get bathy from somewhere (see discussion earlier today).

Need to do the smoothing also.

3) Match parent - child bathy + maskTools/mfiles/mtools/create_nested_grid.m(again)Run this again, but this time set :

merge_par_child_batrhy=1(this calls parentchild_bathy.m)

merge_par_child_mask=1(this calls parentchild_mask.m)You might want to make copies of the grids before you run this again, as this will modify the masking and bathy in both files.4) 3D init and climatologyCOAWST/Tools/mfiles/roms_clm/roms_master_climatology_coawst_mw.m5) Surface forcingsCan use same surface forcing files as parent (or make new ones) 6) roms input file

this happens to be for a 5 grid applicationFirst grid is Lm is 2 points less than total lengthFor all other grids, Lm is 7 points less than total length.Same for Mm.N and Nbed and tracers are same for all gridscan tile differently,but need same total for each gridtime step needs to divide evenly into parent6) roms input file

6) roms input file

6) roms input file

6) roms input file

6) roms input file

7) coawst.bash

Set NestedGrids to be the TOTAL number of grids8) run it

Set np = number of procs,same for each grid- How does the coupled modeling system work?

and

- Setting up a coupled application

20Model Coupling ToolkitMathematics and Computer Science Division Argonne National Laboratoryhttp://www-unix.mcs.anl.gov/mct/

MCT is an open-source package that provides MPI based communications between all nodes of a distributed memory modeling component system. Download and compile as libraries that are linked to.Warner, J.C., Perlin, N., and Skyllingstad, E. (2008). Using the Model Coupling Toolkit to couple earth system models. Environmental Modeling and SoftwareCoupled Modeling SystemModel A running on M nodes.

Model B running on N nodes.Model C (it also works here)MCT providescommunications between all models.LibrariesMCT - v2.60 or higher (distributed)

1) cd to the MCT dir

2) ./configureThis makes Makefile.conf. you can edit this file.

3) make

4) make install

5) set environment varssetenv MCT_INCDIR COAWST/Lib/MCT/includesetenv MCT_LIBDIR COAWST/Lib/MCT/lib(or where ever you installed them, see last slide)

Compilers dir (side note)

Model organizationmaster.Fmpi_initinitrunfinalizeinitrunfinalize{{SWANROMSinit_file (# procs/model)24init, run, and finalizeROMSSWANinit(grid decomp)run(sync. point)finalizeinit_paraminit_parallelinit_scaclarsinit_couplingroms_initroms_runmain3d .....waves_coupling...mpi_finalizeclose_ioroms_finalizeMPI_INITSWINITMPISWMAINSWEXITMPISWINITSWREAD (grid)init_couplingswanmain .....ocean_coupling...mpi_finalizeclose_io25Grid decomposition (during initialization)ROMSSWAN

Each tile is on a separate processor.

Each tile registerswith MCT.26

init_couplingROMS- init_couplingSWAN- init_coupling112233processed by each ROMS tileprocessed by each SWAN tile27

Synchronization (run phase)ROMS- ocean_ouplingSWAN- waves_couplingMCTMCTprocessed by each ROMS tileprocessed by each SWAN tile28Let's look at the fields exchanged between models.ATM - OCN interactionsATMUwind, Vwind, Patm, RH, Tair,cloud, rain, evap, SWrad, LwradLH, HFX, Ustress, Vstress

OCN

Integration and Application Network (ian.umces.edu/symbols),University of Maryland Center for Environmental Science.

Uwind, VwindSwrad, Lwrad,RH, Tair, cloudUstress, Vstress,Swrad, LwradLH, HFXUse momentum + heat fluxes computed in WRF for both ROMS+WRFUse wrf vars in COARE algorithmrain, evapSalt fluxor#define BULK_FLUXES#define ATM2OCN_FLUXES#define ATM_PRESS - Patm#define EMINUSPstflx_temp = Swrad+Lwrad +LH+HFXstflx_salt = evap - rainLH + HFX computed in bulk_fluxes

ATM interactionsOCNATMWAVSSTHwave, Lpwave, Tpsurf,

Surface fluxesMomentumHeatMoisture

= f ( Hwave, Lpwave, Tpsurf )SSTOCNWAVHow to create coupled application1) Create all input, BC, init, forcing, etc files for each model as if running separately. I recommend that you run each model separately first.2) modify cppdefs in your header file.3) SCRIP (if different grids)4) coupling.in5) coawst.bash6) run it as coawstM1) Use each model separatelyWRF27 vertical levelsdt 36 s PhysicsLin microphysicsRRTM longwave, Dudhia shortwaveMellor-Yamada-Janjic (MYJ) PBLKain-Fritsch (KF) cumulus scheme

6 km grid

ROMS16 vertical levelsdt 240, 48PhysicsGLS turbulence closureCOARE bulk fluxesBC's from HYCOMTimestep = 240s

5km and 1 km grid(s)These models are on different grids. 2) south_car.h

3) SCRIP - grid interpolation

Atmosphere model provides heat flux to cover entire ocean grid. SCRIP interpolations weights needed to remap data fields.Ocean model provides higher resolution and coupled response of SST to atmosphere. But the ocean grid is limited in spatial coverage so atmosphere model must combine data from different sources, which can create a discontinuity in the forcing.

http://climate.lanl.gov/Software/SCRIP/Flux conservativeremapping scheme

Ocean grid5 kmAtm Grid6 kmHFLX

10 GFS data

SST

LibrariesSCRIP - v 1.6 (distributed)Used when 2 or more models are not on the same grid.

1) cd to COAWST/Lib/SCRIP/source dir2) edit makefile3) make

3) SCRIPNeed to prepare SCRIP input files by first converting ROMS and WRF grids to a standard netcdf file type that SCRIP likes.

COAWST\Tools\mfiles\mtools\scrip_wrf.m3) SCRIPNeed to prepare SCRIP input files by first converting ROMS and WRF grids to a standard netcdf file type that SCRIP likes.COAWST\Tools\mfiles\mtools\scrip_roms.m

3) SCRIP

SCRIP input file: scrip_ingrid1_file and grid2_file were created with the matlab m files on the last 2 slidesinterp_file1 and interp_file2 will be the new scrip interpolation weights filesNeed to useconservative and fracarea !!run the program as ./scrip 3) SCRIPNeed to run SCRIP for each grid pair.So if you have 1 WRF grid, driving 2 ROMS gridsthen you need 2 sets of weights.

WRF grid 1

1000 m5000 mROMS grid 1

ROMS grid 2

4) coupling.in (this is a ROMS+WRF app)set # procs for each model(total = 56)set coupling interval. for now leave it the same for all models.input file names. only 1 for WRF, 1 for ROMS, multiple for SWANSCRIP weights are listed hereset which WRF grid to couple to4a) ocean.in

set # procs for ocean modelhad listed 20 in coupling.in5 x 4 = 20need dt of 240 to divide evenly into coupling intervalof 1200 sec.4b) namelist.input

set # procs for atm modelhad listed 36 in coupling.in6 x 6 = 36need dt of 30 to divide evenly into coupling intervalof 1200 sec.5) coawst.bash

set # nested roms / swan grids,app name,etc6) run it as coawstMuse total number of procs from coupling.inonly 1 executable

Processor allocation

stdout reports processor allocation

This looks like from a different run,but you get the ideaProcessor allocation"Timing for ." = WRF

"1 179 52974 02:59:00 " = ROMSHere is where the model coupling synchronization occurs.so probably could re-allocate more nodes to WRFJOE_TC - test case examplesJOE_TC test cases are distributed applications for testing ROMS+WRF couplingJOE_TCw = wrf onlyJOE_TCs = same grid, roms + wrf coupledJOE_TCd = different grids for roms and wrf, needs scrip weights