jpolton’s ﬁrst docs documentation · that handle boundary ﬁles if only generating mesh data...

jpolton’s first docs DocumentationRelease 0

jpolton

Apr 17, 2018

Contents

1 Generate AMM60 mesh and mask files from stable build 31.1 Check out v3.6_stable and compile Karen’s code . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Run EXP_meshgen from restart with nn_msh = 1 . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Testing a new compilation with NSea experiment 11

3 AMM60 restarts and XIOS dev 153.1 Clean experiment directory, EXP_SBmoorings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.2 Second Run - debugging XML. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.3 Try and Do these runs in Karen’s compiled code . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.4 Reduce the number of points in the moorings (from 4 to 1). And subsample space . . . . . . . . . . 34

4 Outputting moorings as a masked 3D array 414.1 Create a mask file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.2 Simulation plan for June 2012 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5 Compiling AMM60 45

6 3D Harmonic Analysis 496.1 Make a clean EXPeriment: EXP_harmIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536.2 Try Two Simultaneous experiments: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586.3 First Run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7 Debugging XIOS in AMM7 717.1 Write multiple copies of same variable in single file . . . . . . . . . . . . . . . . . . . . . . . . . . 717.2 Write two copies of same variable in different domains in single file . . . . . . . . . . . . . . . . . . 727.3 AMM7 with two moorings files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 767.4 Next steps: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 787.5 New Plan: Create a mask and output moorings as a 3D array . . . . . . . . . . . . . . . . . . . . . . 817.6 Problems with recompiled code. Start fresh with gmaya copy . . . . . . . . . . . . . . . . . . . . . 84

8 AMM60 runs for ANChor 878.1 New subdomain for North Sea, Jan 2010 - Nov 2013 . . . . . . . . . . . . . . . . . . . . . . . . . . 908.2 Compression of output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 928.3 Unzip jun-aug for 2010-2013 for Locate project . . . . . . . . . . . . . . . . . . . . . . . . . . . . 938.4 Sharing with Matthew Toberman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

9 AMM7 3D Harmonic build and submission 95

i

9.1 0. Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 959.2 1. Copy and run code from Maria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 959.3 2. Compile new executable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 979.4 3. Other things learnt while debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1009.5 4. Identify code changes in Maria’s source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1009.6 5. Check for differences between AMM7 and AMM60 . . . . . . . . . . . . . . . . . . . . . . . . . 1019.7 6. Build harmonic code from clean checkout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

10 Diagnostics of 3D Harmonics 105

11 Introduction 107

12 3D Harmonic analysis 109

13 AMM60 June 2012 restarts and virtual mooring output 111

14 NEMO skills 113

15 ANChor runs 115

16 AMM60 in v3.6_STABLE 117

17 Indices and tables 119

ii

jpolton’s first docs Documentation, Release 0

Pages made:

Contents 1


2 Contents

CHAPTER 1

Generate AMM60 mesh and mask files from stable build

Notes pulled from Evernote. https://www.evernote.com/shard/s523/nl/2147483647/f3d110b9-fcdc-4882-8670-8df042cc949c/ (written 22 Sept 2016)

• Check out v3.6_stable and compile Karen’s code in XIOS_AMM60_nemo

• Run EXP_meshgen from restart with nn_msh = 1

PATH:

/work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r6939/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_→˓meshgen

CONCLUSION:

Can create an AMM60 mesh_mask.nc file from the v3.6_STABLE code using nn_msh = 1 as a restart.

Modules:

module add cray-hdf5-parallelmodule load cray-netcdf-hdf5parallelmodule swap PrgEnv-cray PrgEnv-intel

1.1 Check out v3.6_stable and compile Karen’s code

Then obtain the code:

cd /work/n01/n01/jelt/NEMOsvn co http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE

# Get revision number and relabel the code:

3

https://www.evernote.com/shard/s523/nl/2147483647/f3d110b9-fcdc-4882-8670-8df042cc949c/

https://www.evernote.com/shard/s523/nl/2147483647/f3d110b9-fcdc-4882-8670-8df042cc949c/


less /work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE/.svn/entriesmv nemo_v3_6_STABLE nemo_v3_6_STABLE_r6939

# Get James’ ARCH file:

cp /work/n01/n01/jdha/ST/trunk/NEMOGCM/ARCH/arch-XC_ARCHER_INTEL.fcm /work/n01/n01/→˓jelt/NEMO/nemo_v3_6_STABLE_r6939/NEMOGCM/ARCH/arch-XC_ARCHER_INTEL.fcm

# make the configuration directory (with default options which generates a cfg.txt file with options:XIOS_AMM60_nemo OPA_SRC LIM_SRC_2 NST_SRC):

cd CONFIG./makenemo -n XIOS_AMM60_nemo -m XC_ARCHER_INTEL -j 10

THIS COMPILED OFF THE BAT WITH THE VARIOUS KEYS THAT WERE SELF GENERATED!

# copy in the compiler flags file (Karen’s run). Note, not copying any files from MY_SRC:

cp /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/→˓CONFIG/AMM60smago/cpp_AMM60smago.fcm /work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r6939/→˓NEMOGCM/CONFIG/XIOS_AMM60_nemo/cpp_XIOS_AMM60_nemo.fcm

# Won’t compile because the jdha_init key is not defined without extra code. However, don’t need James’ code,that handle boundary files if only generating mesh data from a cold start. Remove key_jdha_init:

vi XIOS_AMM60_nemo/cpp_XIOS_AMM60_nemo.fcm

# Try and compile again with Karen’s keys:

./makenemo -n XIOS_AMM60_nemo -m XC_ARCHER_INTEL -j 10 clean

./makenemo -n XIOS_AMM60_nemo -m XC_ARCHER_INTEL -j 10

Successful build!

1.2 Run EXP_meshgen from restart with nn_msh = 1

cd /work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r6939/NEMOGCM/CONFIG/XIOS_AMM60_nemo/mkdir EXP_meshgen

Copy run script into job directory:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_NSea/run_nemo EXP_meshgen/run_nemocp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_NSea/submit_nemo.pbs EXP_meshgen/submit_nemo.pbs

Edit run script (Note I only want one restart at this stage):

vi EXP_meshgen/run_nemoexport RUNNAME=EXP_meshgenexport YEARrun='2012'export HOMEDIR=/work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r6939/NEMOGCM/CONFIG/XIOS_→˓AMM60_nemoexport nrestart_max=2 #31 (For one submission this number must equal the number of→˓lines in run_counter.txt) (continues on next page)

4 Chapter 1. Generate AMM60 mesh and mask files from stable build


(continued from previous page)

Edit submission script:

vi EXP_meshgen/submit_nemo.pbs#PBS -N AMM60meshgen

Copy iodef.xml into job directory:

cp /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_notradiff_long/iodef_fastnet.→˓xml EXP_meshgen/iodef.xmlvi EXP_meshgen/iodef.xml

Copy domain_def.xml into job directory:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/domain_def.xml EXP_meshgen/domain_def.xml

Copy finish_nemo.sh into job directory:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/finish_nemo.sh EXP_meshgen/finish_nemo.sh

Link restart files:

mkdir EXP_meshgen/RESTARTln -s /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/AMM60smago/EXPD376/RESTART/01264320 EXP_meshgen/RESTART/.

Create run_counter.txt into job directory (I don’t know the dates. NB Karen’s numbers are quite large but Idon’t see the restart files). Note that the last line 2nd number must be +1 of the restart directory name. BEWARE ofextra white spaces in these lines as the ‘cutting’ will not work properly with them:

vi EXP_meshgen/run_counter.txt1 1 7200 201001052 1264321 1271520

Copy in namelists:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_NSea/namelist_ref EXP_meshgen/.cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_NSea/namelist_cfg EXP_meshgen/.

Edit namelist to generate mesh files:

vi EXP_meshgen/namelist_cfgnn_msh = 1 ! create (=1) a mesh file or not (=0)

Submit run:

cd /work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r6939/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_→˓meshgen./run_nemo3953845.sdb

(continues on next page)

1.2. Run EXP_meshgen from restart with nn_msh = 1 5



sdb:Req'd Req'd Elap

Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----3953845.sdb jelt standard AMM60meshg -- 92 220 -- 00:05 Q -- nn_→˓msh =1. HOW ARE THE MESH FILES?

Job crashed. However ‘‘mesh_mask*.nc‘‘ files contain grid information and is OK:

module load cray-netcdfncdump -h mesh_mask_0208.nc...double e3t_0(t, z, y, x) ;double e3w_0(t, z, y, x) ;

What went wrong: grep “E R R” WDIR/ocean.output:

jelt@eslogin005:/work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r6939/NEMOGCM/CONFIG/XIOS_→˓AMM60_nemo/EXP_meshgen/WDIR> less ocean.output

===>>> : E R R O R===========

E R R O R : misspelled variable in namelist nambdy in reference namelist iostat = 19

===>>> : E R R O R===========

E R R O R : misspelled variable in namelist nambdy in configuration namelistiostat = 19

nambdyNumber of open boundary sets : 1

------ Open boundary data set 1 ------Boundary definition read from file coordinates.bdy.nc

This crash looks solvable. . .

However can I rebuild the mesh files and not bother?:

cd /work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r6939/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_→˓meshgen/WDIR/work/n01/n01/jelt/gmaya/NEMO/TOOLS/jREBUILD_NEMO/rebuild_nemo -t 40 -c 100 mesh_mask→˓2000

Put this command into the serial queue with a 20 minute queue (5mins too short):

cd /work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r6939/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_→˓meshgenqsub nc_mesh_build.pbs

This worked!

• 1st attempt had a problem, but it had to write some list files and terminated quite quickly (2sec)

• 2nd attempt exceeded 5 min wall time.

• 3rd attempt ,with 20mins. Took 4mins. No errors.



Copy output data to SAN and sanity check in Ferret:

[livmaf:/scratch/jelt/tmp] scp [email protected]:/work/n01/n01/jelt/NEMO/nemo_→˓v3_6_STABLE_r6939/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_meshgen/WDIR/mesh_mask.nc .yes? use mesh_mask.ncyes? shade /k=10 E3T_0...yes? sh dacurrently SET data sets:1> ./mesh_mask.nc (default)name title I J K L M→˓ NNAV_LON 1:1120 1:1440 ... ... ...→˓ ...NAV_LAT 1:1120 1:1440 ... ... ...→˓ ...NAV_LEV ... ... 1:51 ... ...→˓ ...TIME_COUNTER

... ... ... 1:1 ...→˓ ...TMASK 1:1120 1:1440 1:51 1:1 ...→˓ ...UMASK 1:1120 1:1440 1:51 1:1 ...→˓ ...VMASK 1:1120 1:1440 1:51 1:1 ...→˓ ...FMASK 1:1120 1:1440 1:51 1:1 ...→˓ ...TMASKUTIL

1:1120 1:1440 ... 1:1 ...→˓ ...UMASKUTIL

1:1120 1:1440 ... 1:1 ...→˓ ...VMASKUTIL

1:1120 1:1440 ... 1:1 ...→˓ ...FMASKUTIL

1:1120 1:1440 ... 1:1 ...→˓ ...GLAMT 1:1120 1:1440 ... 1:1 ...→˓ ...GLAMU 1:1120 1:1440 ... 1:1 ...→˓ ...GLAMV 1:1120 1:1440 ... 1:1 ...→˓ ...GLAMF 1:1120 1:1440 ... 1:1 ...→˓ ...GPHIT 1:1120 1:1440 ... 1:1 ...→˓ ...GPHIU 1:1120 1:1440 ... 1:1 ...→˓ ...GPHIV 1:1120 1:1440 ... 1:1 ...→˓ ...GPHIF 1:1120 1:1440 ... 1:1 ...→˓ ...E1T 1:1120 1:1440 ... 1:1 ...→˓ ... (continues on next page)




E1U 1:1120 1:1440 ... 1:1 ...→˓ ...E1V 1:1120 1:1440 ... 1:1 ...→˓ ...E1F 1:1120 1:1440 ... 1:1 ...→˓ ...E2T 1:1120 1:1440 ... 1:1 ...→˓ ...E2U 1:1120 1:1440 ... 1:1 ...→˓ ...E2V 1:1120 1:1440 ... 1:1 ...→˓ ...E2F 1:1120 1:1440 ... 1:1 ...→˓ ...FF 1:1120 1:1440 ... 1:1 ...→˓ ...MBATHY 1:1120 1:1440 ... 1:1 ...→˓ ...MISF 1:1120 1:1440 ... 1:1 ...→˓ ...ISFDRAFT 1:1120 1:1440 ... 1:1 ...→˓ ...HBATT 1:1120 1:1440 ... 1:1 ...→˓ ...HBATU 1:1120 1:1440 ... 1:1 ...→˓ ...HBATV 1:1120 1:1440 ... 1:1 ...→˓ ...HBATF 1:1120 1:1440 ... 1:1 ...→˓ ...GSIGT ... ... 1:51 1:1 ...→˓ ...GSIGW ... ... 1:51 1:1 ...→˓ ...GSI3W ... ... 1:51 1:1 ...→˓ ...ESIGT ... ... 1:51 1:1 ...→˓ ...ESIGW ... ... 1:51 1:1 ...→˓ ...E3T_0 1:1120 1:1440 1:51 1:1 ...→˓ ...E3U_0 1:1120 1:1440 1:51 1:1 ...→˓ ...E3V_0 1:1120 1:1440 1:51 1:1 ...→˓ ...E3W_0 1:1120 1:1440 1:51 1:1 ...→˓ ...RX1 1:1120 1:1440 ... 1:1 ...→˓ ...GDEPT_1D ... ... 1:51 1:1 ...→˓ ...GDEPW_1D ... ... 1:51 1:1 ...→˓ ...GDEPT_0 1:1120 1:1440 1:51 1:1 ...→˓ ...GDEPW_0 1:1120 1:1440 1:51 1:1 ...→˓ ... (continues on next page)




CONCLUSION:

Can create an AMM60 mesh_mask.nc files from the v3.6_STABLE code using nn_msh = 1 as a restart.


CHAPTER 2

Testing a new compilation with NSea experiment

Having first followed Compiling AMM60 to make a new executable Construct a EXP_NSea experiment to test newbuild.

Update directory with missing files to get it working:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemorsync -uartv /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea_prelauch_copy/→˓ ../XIOS_AMM60_nemo

mkdir EXP_NSea

Edit run_counter to start from restart on the 14th Jan 2010:

vi EXP_NSea/run_counter.txt1 1 7200 201001052 7201 14400 7200=201001093 14401 21600 14400=20100114

Note this copies old memo.exe and xios_server.exe into the config directory. But these are replaced / updated by therun_nemo script

Edit run_nemo:

vi run_nemoexport HOMEDIR= /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/XIOS_AMM60_nemo # Home Directory...export XIOSDIR=/work/n01/n01/jdha/ST/xios-1.0/

cp XIOS_AMM60_nemo/submit_nemo.pbs XIOS_AMM60_nemo/EXP_NSea/submit_nemo.pbsvi XIOS_AMM60_nemo/EXP_NSea/submit_nemo.pbs

Set a 5 min wall time.

Copy in namelist files:

11

Compiling_AMM60.html


cd XIOS_AMM60_nemocp namelist_ref EXP_NSea/.cp namelist_cfg EXP_NSea/.

cp iodef.xml EXP_NSea/.cp domain_def.xml EXP_NSea/.

Add in restart files:

ln -s /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_notradiff_long/RESTART/→˓00014400 /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_NSea/RESTART/00014400

Submit:

./run_nemo3941432.sdb

IT RUN AND WAS KILLED BY THE WALL TIME LIMIT.

Housecleaning: move unnecessary files in /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo to /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/TMP

Change the queue to 20mins.run_counter.txt OK.To check TMP storage OK.

Resubmit:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_NSea./run_nemo3941652.sdb


Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----3941652.sdb jelt standard AMM60_run -- 92 220 -- 00:20 Q --

IT WORKS

(16 Sept 2016)

Check output: AMM60_read_plot.ipynb Looks OK. Though has a non zero mean in this subdomain. Plot zos -SSH, hourly

NEW BUILD /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_NSea/OUTPUT/AMM60_1h_20100115_20100119_NorthSea1.nc

OLD BUILD (Karen’s executable) /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUT/AMM60_1h_20100115_20100119_NorthSea1.nc

12 Chapter 2. Testing a new compilation with NSea experiment


Old build and new build look the same. Hooray!

13


14 Chapter 2. Testing a new compilation with NSea experiment

CHAPTER 3

AMM60 restarts and XIOS dev

Aim to pick up and old 2012 restart file and run with new iodef.xml output.

Copy the restart file from Karens run. Perhaps /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/AMM60smago/EXPD376

In particular I want to rerun the June 2012 period (when there was a cruise) and output lots of virtual moorings for JoHopkins’ student (David) to analyse the shelf break processes.

NOTE:

Outputting 3305 moorings using XIOS 1d techniques failed. Tried distributing moorings across different numbers offiles. Tried varying the processor allocation to XIOS. There were either walltime issues or memory problems. Thismight be fixed in xios-2. . . In the end outputted the data as a masked 3D spatial array, though not a elegant it was fastand worked!

PLAN:

1. Create a clean experiment directory

2. Clean up the iodef.xml file

3. Test run a few days

4. Expand the iodef.xml file

5. Do one month simulation

NOTE I DO NOT KNOW THE DATE OF THE RESTART: Update 1271520=20120605 1264321 Seems to correspondto midnight 1 June 2012 (plus or minus 30 mins)

3.1 Clean experiment directory, EXP_SBmoorings

Follow notes from Testing a new compilation with NSea experiment:

15

Testing_a_new_compilation_with_NSea_experiment.html

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemomkdir EXP_SBmoorings

Copy run script into job directory. Note this is not necessary as all the paths are set in the script but is means that eachexperiment can have its own run script:

cp EXP_NSea/run_nemo EXP_SBmoorings/run_nemocp EXP_NSea/submit_nemo.pbs EXP_SBmoorings/submit_nemo.pbs

Edit run script (Note for only one restart restart_max=2):

vi EXP_SBmoorings/run_nemoexport RUNNAME=EXP_SBmooringsexport YEARrun='2012'export HOMEDIR=/work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/XIOS_AMM60_nemoexport nrestart_max=7 (For one submission this number must equal the number of lines→˓in run_counter.txt)

vi EXP_SBmoorings/submit_nemo.pbs#PBS -N AMM60_SB#PBS -l walltime=00:20:00..

$JOBDIR/finish_nemo.sh$JOBDIR/run_nemo

..


cp /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_notradiff_long/iodef_fastnet.→˓xml EXP_SBmoorings/iodef_sbmoorings.xmlvi iodef_sbmoorings.xml (E.g. for testing)

<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE."> ...

<file id="file001" name_suffix="_SB001_grid_T" description="ocean T grid→˓variables">

<field_group id="1h_SB001_grid_T" domain_ref="SB001" ><field field_ref="toce" name="thetao" long_name="sea water

→˓potential temperature" /><field field_ref="soce" name="so" long_name="sea water salinity

→˓" /><field field_ref="tpt_dep" name="depth" /><field field_ref="uoce" name="vozocrtx" long_name="sea water x

→˓velocity" /><field field_ref="voce" name="vomecrty" long_name="sea water y

→˓velocity" /><field field_ref="woce" name="vovecrtz" long_name="sea water w

→˓velocity" /><field field_ref="utau" name="utau" long_name="surface downward x

→˓stress" /><field field_ref="vtau" name="vtau" long_name="surface downward y

→˓stress" /><field field_ref="ssh" name="ssh" long_name="sea surface

→˓height"/></field_group>


16 Chapter 3. AMM60 restarts and XIOS dev

</file>

</file_group>


cp /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/domain_def.xml EXP_→˓SBmoorings/domain_def.xml

Edit XML files to contain virtual moorings at given lat/lon locations from supplied list. Use python notebook togenerate text to paste into XML files: jcomp_tools_dev/AMM60_build_iodef_from_latlon.ipynb:

vi domain_def.xml (E.g. for testing)

<domain id="SB032" zoom_ibegin="0610" zoom_jbegin="0302" zoom_ni="2" zoom_nj="2

→˓" /><domain id="SB001" zoom_ibegin="0608" zoom_jbegin="0304" zoom_ni="2" zoom_nj="2




















→˓" />(continues on next page)

3.1. Clean experiment directory, EXP_SBmoorings 17



<domain id="SB021" zoom_ibegin="0614" zoom_jbegin="0327" zoom_ni="2" zoom_nj="2→˓" />











When happy overwrite the iodef.xml with the new version:: cp EXP_SBmoorings/iodef_sbmoorings.xmlEXP_SBmoorings/iodef.xml

Copy finish_nemo.sh into job directory:: cp /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/finish_nemo.shEXP_SBmoorings/finish_nemo.sh

Link restart files:

mkdir EXP_SBmoorings/RESTARTln -s /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/AMM60smago/EXPD376/RESTART/01264320 EXP_SBmoorings/RESTART/.

Create run_counter.txt into job directory (I don’t know the dates. NB Karen’s numbers are quite large but Idon’t see the restart files). Note that the last line 2nd number must be +1 of the restart directory name. BEWAREof extra white spaces in these lines as the ‘cutting’ will not work properly with them Edit run_counter.txt:1264321 is 1st June 2012, or perhaps 20120531 as this appears in the output file:

Chopped from AMM60_SB.o396004101271520

date ndastp : 20120531

Anyway edit run_counter.txt to start at the beginning on June 2012:

vi EXP_SBmoorings/run_counter.txt1 1 7200 201001052 1264321 1271520

Copy in namelists:

cp EXP_NSea/namelist_ref EXP_SBmoorings/.cp EXP_NSea/namelist_cfg EXP_SBmoorings/.

Submit run:



cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings./run_nemo3971205.sdb


Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----3971205.sdb jelt standard AMM60_SB -- 92 220 -- 00:20 Q -- <--→˓IN PROGRESS. CAREFUL WALL TIME MAY BE EXCEEDED BY LARGE NUMBERS OF OUTPUT FILES→˓(3305 moorings).

IT BROKE. TRY JUST ONE SUBMISSION TO DEBUG. ALSO cut down the number of output files iniodef.xml.

vi run_counter.txt1 1 7200 201001052 1264321 1271520

vi run_nemoexport nrestart_max=2 #31 (For one submission this number must equal the number of→˓lines in run_counter.txt)

Shorten the queue to get this thing going (hopefully):

vi submit_nemo.pbs#PBS -l walltime=00:01:00

Clean up a bit:

rm -r OUTPUT/ WDIR/ LOGS/

Cut down the number of XML output files to file000 - file999 in iodef.xml.Original list is in iodef_sbmoorings.xmlIf this work I will need to run the month 3 times to simulate 3305 moorings.

Submit run:



Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----3972357.sdb jelt standard AMM60_SB -- 92 220 -- 00:01 Q --

Broke. Looks like it didn’t like the iodef.xml fileSave iodef.xml with 1000 virtual moorings as iodef_sbmoorings_000_999.xml


Recover simple iodef.xml file from ANChor run:

cp /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/iodef.xml iodef.xml

Clean it up and fix it to have nothing but the following:

<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE."> <file id="file011" name_suffix="_SB011_grid_T" description="ocean T grid variables"><field_group id="1h_SB011_grid_T" domain_ref="SB011" >

<field field_ref="toce" name="thetao" long_name="sea water potential→˓temperature" />

<field field_ref="soce" name="so" long_name="sea water salinity"→˓ />

<field field_ref="tpt_dep" name="depth" /><field field_ref="uoce" name="vozocrtx" long_name="sea water x velocity" /

→˓><field field_ref="voce" name="vomecrty" long_name="sea water y velocity" /

→˓><field field_ref="woce" name="vovecrtz" long_name="sea water w velocity" /

→˓><field field_ref="utau" name="utau" long_name="surface downward x stress

→˓" /><field field_ref="vtau" name="vtau" long_name="surface downward y stress

→˓" /><field field_ref="ssh" name="ssh" long_name="sea surface height"/>

</field_group></file>

</file_group>

Submit run with 20min wall time:




PENDING. Does it produce output? Yes, AMM60_1h_20120601_20120605_SB011_grid_T.nc exists. Itis running now (15:55, 4 Oct 2016)

Yes. Spotted error in the iodef_sbmooring*.xml files. Double definition of the 1h file_group without closingit. <file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE."> 

Saved the working test iodef files: iodef_1mooring.xmlCopied the full file to the operational iodef file: cp iodef_sbmoorings_001_3305.xml iodef.xml

Trim run_counter.txt

Resubmit:





PENDING. Does it produce mooring output?CAREFUL WALL TIME MAY BE EXCEEDED BY LARGE NUMBERS OF OUTPUT FILES (3305 moorings). 7Oct 201

EXP_SBmoorings/LOGS/01271520> less stdouterr-> report : Memory report : Context <nemo> : client side : total memory used for→˓buffer 0 bytes

Try and rewrite the XML output to all be in one file.Create a separate lookup for lat and lon.Save new file as iodef_1file.xml


cp iodef_1file.xml iodef.xml

Check the 20min queue

Resubmit:




<— PENDING. Does it produce mooring output?CAREFUL WALL TIME MAY BE EXCEEDED BY LARGE NUMBERS OF VARIABLES in FILE (3305moorings). 10 Oct 2016



11 Oct

It runs and adds to run_counter.txt:

1 1 7200 201001052 1264321 12715203 1271521 1278720 1271520=20120605

Though OUTPUT contains no new files. (Though it should all go in one file now):

module load cray-netcdfncdump -h OUTPUT/*nc

``time.step: 1271520`` -- indicates the run properly finished integration

less AMM60_SB.o3982808 -- likewise shows wall time was not exceeded

cd EXP_SBmoorings/LOGS/01271520less time-step ocean.output_EXP_SBmoorings

Some warnings but no errors.Presumably a problem with the iodef.xml file

Action: Check the iodef.xml file

Copy iodef.xml to give a local file for inspection:

cp iodef.xml ~/Desktop/OneWeekExpiry/.

Cut the file down to just a few field_group entries. Resubmit:




<— PENDING. Does it produce mooring output? (11 Oct 2016)EXPECT A SINGLE MOORING *.nc FILE. WITH A NUMBER OR MOORINGS WITHIN.

12 Oct 2016

It ran run_counter.txt has next step readyOne nc output file, which is old. So no new XML output!Finished fine in 16mins. No walltime problem


Nothing wrong in LOGS/01271520/ocean.output_EXP_SBmooringsiodef.xml file is OK.

Action Need to debug XML file in AMM7, on the short queue.

Spotting a spurious quote mark “ at the end of file_group definition:

<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE."> "

Try the whole lot in one go:

cp iodef_1file.xml iodef.xml


Check 20min queue

Resubmit:


Meanwhile remove spurious quotation mark ” in iodef_sbmoorings_001_3305.xml andiodef_sbmoorings_001_999.xml

PENDING (18 Oct 2016)

NO NETCDF OUTPUT. Needs further investigation.

Does it work with two moorings?:

cp iodef_2moorings.xml iodef.xml


Check 20min queue

Resubmit:


IT WORKS : 5 days came in at 20 mins, with two moorings. Nearly hit walltime. Renamed OUTPUT/AMM60*ncfile to ..SB2.nc, or something similar.

Next step try 1000 moorings. . .


cp iodef_sbmoorings_001_999.xml iodef.xml


Check 20min queue

Resubmit:


PENDING (19 Oct 2016) COMPLETED late, 10pm. Need to look at data.

• Is there lots of mooring output in a single file? AMM60_1h_20120601_20120605_SB_grid_T.nc iscreated. Wall time exceeded. 27 hours completed.

• Is the output from the prevous run, with two moorings, OK? - These data have 5 days of data

• Could try a few files with multiple moorings in each, say 33 files 100 moorings?

cp iodef_sbmoorings_33files.xml iodef.xml

Trim run_counter.txt. Not neededCheck 20min queue. OK

Resubmit:



• Are there 33 files of 100 moorings?

• Is the data OK?

• How is the data in the iodef_sbmoorings_001_999.xml simulation?

There are NO XML outputsNasty garbled run_counter.txt data, though it looks like it finished as a 3rd line is addedNothing new in OUTPUTRan for 10 mins (20min wall time)

Disparity between time.step and run_counter.txt:



more WDIR/time.step1265827

more run_counter.txt1 1 7200 201001052 1264321 12715203 1271521 1278720 _:U()I.=

Check LOGS:

more stdouterr[NID 04371] 2016-10-19 22:21:27 Apid 23764253: initiated application termination[NID 04371] 2016-10-19 22:21:28 Apid 23764253: OOM killer terminated this process.Application 23764253 exit signals: KilledApplication 23764253 resources: utime ~0s, stime ~68s, Rss ~4848, inblocks ~4163,→˓outblocks ~166

Out of memory:http://www.nersc.gov/users/computational-systems/retired-systems/hopper/running-jobs/→˓memory-considerations/Recommend use fewer processors on each node, and therefor more nodes

How for did the run get? 25.1 hours Surely the output should have started appearing?

Found bug in iodef_sbmoorings_33files.xml: the variables names were not unique in the files. Fixed this.Resubmit with more memory on the XIOS nodes.

cp iodef_sbmoorings_33files.xml iodef.xml

Configure AMM60 SBmoorings to run on more XIOS nodes

Standard:

submit_nemo.pbs:#PBS -l select=92export NEMOproc=2000export XIOSproc=40aprun -b -n $NEMOproc -N 24 ./$EXEC : -N 5 -n $XIOSproc ./xios_server.exe >&stdouterr

vi run_nemoexport NPROC=2000

Sums:NEMO nodes: ceil(2000 / 24) = 84XIOS nodes: ceil(40 / 5) = 8Total = 92

New Double XIOS nodes:

submit_nemo.pbs:#PBS -l select=**100**export NEMOproc=2000export XIOSproc=**80**aprun -b -n $NEMOproc -N 24 ./$EXEC : -N 5 -n $XIOSproc ./xios_server.exe >&stdouterr








Resubmit:




• How is the speed up with twice as many XIOS processors?

Something Broke. Looks like a problem with the file naming, though file011 has previously worked Edit 34 file namesto be called file101 - file134. These edits are in iodef.xml and not in iodef_sbmoorings_33files.xml.

Not sure what to do! Need to download logs.

Looking at the logs:

6MB file created: AMM60_1h_20120601_20120605_SB001_grid_T.nc –> Can not read with ncdump or ferret.HDF error.run_counter.txt gets extra line though the data string is garbled.ran for 7 minscore dumptime.step : 1264632 –> 5 hours of model time integration

less LOGS/restart/stdouterr

terminate called after throwing an instance of 'terminate called after throwing an→˓instance of 'xios::CNetCdfExceptionxios::CNetCdfException''

what(): Error in calling function nc_enddef(ncId)NetCDF: HDF errorUnable to end define mode of this file, given its id : 65536

terminate called after throwing an instance of 'xios::CNetCdfException'what(): Error in calling function nc_enddef(ncId)

NetCDF: HDF errorUnable to end define mode of this file, given its id : 65536

forrtl: error (76): Abort trap signal

This seems consistent with running out of memory.

Try new configuration of XIOS processors



Standard:




New Double XIOS nodes:

submit_nemo.pbs:#PBS -l select=**104**export NEMOproc=2000export XIOSproc=**60**aprun -b -n $NEMOproc -N 24 ./$EXEC : -N **3** -n $XIOSproc ./xios_server.exe >&→˓stdouterr



trim run_counter.txt

Resubmit (Note I killed the original submission to fix the namelist_cfg sed edit issue with nn_write not be changed):





run_counter.txt with new line but garbled dateOUTPUT/AMM60_1h_20120601_20120605_SB001_grid_T.nc exists but can not be read by either ncdumpor FERRET.less time.step: 1264692



wall time of 6mins before terminating

LOGS/restart:

vi stdouterr



terminate called after throwing an instance of 'terminate called after throwing an→˓instance of 'xios::CNetCdfExceptionxios::CNetCdfException''



forrtl: error (76): Abort trap signalImage PC Routine Line Sourcexios_server.exe 0000000000CB57E1 Unknown Unknown Unknownxios_server.exe 0000000000CB3F37 Unknown Unknown Unknown...

3.2 Second Run - debugging XML.

Make new EXPeriment:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/mkdir EXP_SBmoorings2

Copy files but not directories:

cp EXP_SBmoorings/* EXP_SBmoorings2/.

Link restart files:

mkdir EXP_SBmoorings2/RESTARTln -s /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/AMM60smago/EXPD376/RESTART/01264320 /work/n01/n01/jelt/NEMO/NEMOGCM_→˓jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings2/→˓RESTART/.

Edit run_counter (run for 24 hours):

cd EXP_SBmoorings2vi run_counter.txt



1 1 7200 201001052 1264321 1265760

Edit submission script, and maybe the wall time:

vi submit_nemo.pbs#PBS -N AMM60_SB2#PBS -l walltime=00:20:00

Edit run file for new directory path:

vi run_nemoexport RUNNAME=EXP_SBmoorings2

Edit iodef.xml file to have 100 moorings in one file and 5 in the second (last) file:

less iodef_sbmoorings_100moorings_2files.xml

<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE."> <file id="file001" name_suffix="_SB001_grid_T" description="ocean T grid

→˓variables"><field_group id="1h_SB001_grid_T" domain_ref="SB001" ><field field_ref="toce" name="thetao_SB001" long_name="sea water

→˓potential temperature" /><field field_ref="soce" name="so_SB001" long_name="sea water

→˓salinity" />...<field field_ref="vtau" name="vtau_SB100" long_name="surface

→˓downward y stress" /><field field_ref="ssh" name="ssh_SB100" long_name="sea surface


</file><file id="file034" name_suffix="_SB034_grid_T" description="ocean T grid






</file></file_group>

cp iodef_sbmoorings_100moorings_2files.xml iodef.xml

Resubmit (Note I killed the original submission to fix the namelist_cfg sed edit issue with nn_write not beingchanged):

3.2. Second Run - debugging XML. 29



PENDING (21 Oct 2016) cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings2

• Are there 1 file of 100 moorings and 1 files of 5 moorings?


OUTPUT/AMM60_1h_20120601_20120601_SB001_grid_T.nc exists but is not readable by Ferret orncdump

run_counter.txt completed but new date is garbledwall time of 2mins was usedless time.step: 1264871

core dump

In the LOGS/restart:

less stdouterr



forrtl: error (76): Abort trap signalImage PC Routine Line Sourcexios_server.exe 0000000000CB57E1 Unknown Unknown Unknownxios_server.exe 0000000000CB3F37 Unknown Unknown Unknown

3.3 Try and Do these runs in Karen’s compiled code

Start with old directory:

cd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smagomkdir SBmoorings3

Copy files but not directories:

cp EXP_NSea/* EXP_SBmoorings3/.

Link restart files:

mkdir EXP_SBmoorings3/RESTARTln -s /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/AMM60smago/EXPD376/RESTART/01264320 EXP_SBmoorings3/RESTART/.

Edit run_counter (run for 24 hours):



cd EXP_SBmoorings3vi run_counter.txt1 1 7200 201001052 1264321 1265760

Edit submission script, and maybe the wall time:

vi submit_nemo.pbs#PBS -N AMM60_SB3#PBS -l walltime=00:20:00

Edit run file for new directory path:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/run_nemo .vi run_nemoexport RUNNAME=EXP_SBmoorings3..export HOMEDIR=/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago

Check max restarts too. Note where field_def.xml is copied from.

Copy the other XML files:

mkdir /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/SHAREDcp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓SHARED/field_def.xml /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/SHARED/.cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/domain_def.xml .

Edit iodef.xml file to have 100 moorings in one file and 5 in the second (last) file:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings2/iodef_sbmoorings_100moorings_2files.xml .less iodef_sbmoorings_100moorings_2files.xml


→˓files --><file id="file001" name_suffix="_SB001_grid_T" description="ocean T grid








→˓potential temperature" />(continues on next page)

3.3. Try and Do these runs in Karen’s compiled code 31



<field field_ref="soce" name="so_SB3301" long_name="sea water→˓salinity" />

...<field field_ref="vtau" name="vtau_SB3305" long_name="surface




cp iodef_sbmoorings_100moorings_2files.xml iodef.xml

Resubmit (Note I killed the original submission to fix the namelist_cfg sed edit issue with nn_write not beingchanged):


PENDING (21 Oct 2016) cd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_SBmoorings3

stopped in 21score dumpempty OUTPUT/less LOGS/restart/stdouterr: Can not open <./field_def.xml> file

Fix missing field_def.xml (edited instructions above)Trim run_counter.txt

Resubmit:





killed. Wall time exceeded

Extend wall time, check run_counter.txt and resubmit:










killed. Wall time exceeded

Extend wall time, check run_counter.txt and resubmit:

vi run_counter.txt # This is one day1 1 7200 201001052 1264321 1265760






Wall time exceeded, 1h 30. This is for a 1 day simulation!

Configure AMM60 SBmoorings to run on more XIOS nodes

Standard:




New x10 XIOS nodes:

3.3. Try and Do these runs in Karen’s compiled code 33


submit_nemo.pbs:#PBS -l select=**164**export NEMOproc=2000export XIOSproc=**400**aprun -b -n $NEMOproc -N 24 ./$EXEC : -N 5 -n $XIOSproc ./xios_server.exe >&stdouterr




Resubmit to a 20min queue:

cd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_SBmoorings3./run_nemo4007516.sdb

FAILED (23 Oct 2016)

• Are there 33 files of 100 moorings + 5 in another?


It completed in 17s!

stdouterr: apsched: claim exceeds reservation's resources

3.4 Reduce the number of points in the moorings (from 4 to 1). Andsubsample space

Revert back to my compiled code:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings

Manually edit domain_def.xml (1pt per mooring). LATER edit and iodef.xml (moorings divisible by 4):

cp domain_def.xml domain_def_4pt.xmlmv domain_def.xml domain_def_1pt.xml

vi domain_def_1pt.xml:12,3318s/zoom_ni="2" zoom_nj="2"/zoom_ni="1" zoom_nj="1"/g

cp domain_def_1pt.xml domain_def.xml

Since 30 moorings with 4pts worked in the test phase, then 100 moorings should work with 1pt:



cp ../EXP_SBmoorings2/iodef_sbmoorings_100moorings_2files.xml ../EXP_SBmoorings/iodef_→˓sbmoorings_100moorings_2files.xmlcp iodef_sbmoorings_100moorings_2files.xml iodef.xml

Edit run_counter.txt for 5 day simulation:


Extended XIOS processors (20 mins wall time):

vi submit_nemo.pbs#PBS -l select=104#PBS -l walltime=00:20:00export NEMOproc=2000export XIOSproc=60aprun -b -n $NEMOproc -N 24 ./$EXEC : -N 3 -n $XIOSproc ./xios_server.exe >&stdouterr

Resubmit:


cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings


• Are there 2 files with 100 moorings in one + 5 in another?

• How is the speed?

NOT WORKING.There is only one file: OUTPUT/AMM60_1h_20120601_20120605_SB001_grid_T.nc and it is broken.core dumpran for 2min 33less time.step: 1264875completed with garbled new datestamp on run_counter.txtless stdouterr:

:: terminate called after throwing an instance of ‘xios::CNetCdfException’ what(): Error in calling functionnc_enddef(ncId) NetCDF: HDF error Unable to end define mode of this file, given its id : 65536

forrtl: error (76): Abort trap signal Image PC Routine Line Source xios_server.exe 0000000000CB57E1 Un-known Unknown Unknown xios_server.exe 0000000000CB3F37 Unknown Unknown Unknown

PERHAPS THERE IS A BUG WITH THE SECOND XML OUTPUT FILE.

ACTION: DEBUG XML in AMM7.

AMM7 suggests that there are too many moorings per file.AMM7 configuration works with 25 moorings per file. Also 4pt per mooring increases run time by around 16%.

Try 1pt moorings. 25 moorings in 133 files.

3.4. Reduce the number of points in the moorings (from 4 to 1). And subsample space 35


cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmooringscp domain_def_1pt.xml domain_def.xmlcp /work/n01/n01/jelt/gmaya/NEMO/CONFIG/XIOS_AMM7_nemo2/EXP00/iodef_sbmoorings_→˓25moorings_133files.xml . # Get the new iodef filecp iodef_sbmoorings_25moorings_133files.xml iodef.xml

Trim run_counter.txt:: vi run_counter.txt # This is one day 1 1 7200 20100105 2 1264321 1265760

20min queue. Resubmit:: ./run_nemo 4015266.sdb

WALLTIME EXCEEDED

Resubmit with only 10 output files. Check run_counter.txt:: ./run_nemo 4016256.sdb

WALLTIME EXCEEDED

3.4.1 PLAN: OUTPUT 3D data with a SBmooring mask.

NEED TO EDIT AND RECOMPILE THE CODE

Copy the modifications to the executable:

cp /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG/XIOS_AMM7_nemo/MY_SRC/→˓diawri.F90 /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/XIOS_AMM60_nemo/MY_SRC/.

Edit field_def.xml to SHAREDDIR:

**CHANGE FIELD_REF to ID**vi /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓SHARED/field_def.xml

<field_group id="sbmooring" ><field field_ref="sb_toce" name="thetao" long_name="sea_water_potential_

→˓temperature" unit="degC" grid_ref="grid_T_3D" /><field field_ref="sb_soce" name="so" long_name="sea_water_salinity"

→˓ unit="psu" grid_ref="grid_T_3D" /><field field_ref="sb_u" name="uo" long_name="sea_water_x_velocity"

→˓ unit="m/s" grid_ref="grid_U_3D" /><field field_ref="sb_v" name="vo" long_name="sea_water_y_velocity"

→˓ unit="m/s" grid_ref="grid_V_3D" /><field field_ref="sb_w" name="wo" long_name="sea_water_z_velocity"

→˓ unit="m/s" grid_ref="grid_W_3D" /><field field_ref="sb_dept" name="depth" long_name="T-cell thickness"

→˓ unit="m" grid_ref="grid_T_3D" /><field field_ref="sb_ssh" name="zos" long_name="sea_surface_height_

→˓above_geoid" unit="m" grid_ref="grid_T_2D" /><field field_ref="sb_utau" name="tauuo" long_name="surface_downward_x_

→˓stress" unit="m/s^2" grid_ref="grid_U_2D" /><field field_ref="sb_vtau" name="tauvo" long_name="surface_downward_y_

→˓stress" unit="m/s^2" grid_ref="grid_V_2D" /></field_group>

Edit iodef.xml in JOBDIR::

vi iodef.xml <!– Shelf Break virtual moorings –> <file_group id=”1h” output_freq=”1h” output_level=”10”enabled=”.TRUE.”> <!– 1h files –>



<file id=”file51” name_suffix=”SB” description=”Shelf break moorings”> <field_groupgroup_ref=”sbmoorings”/>

</file> ..

</file_group>

Recompile

load modules:


Compile:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG./makenemo -n XIOS_AMM60_nemo_harmIT -m XC_ARCHER_INTEL -j 10

Copy executable (note that this is for 3D harmonics, though it is not invoked here):: mvXIOS_AMM60_nemo/EXP_SBmoorings/nemo.exe XIOS_AMM60_nemo/EXP_SBmoorings/nemo.exe301016cp XIOS_AMM60_nemo_harmIT/BLD/bin/nemo.exe XIOS_AMM60_nemo/EXP_SBmoorings/.

Trim run_counter.txt:

cd XIOS_AMM60_nemo/EXP_SBmoorings/vi run_counter.txt

Hit wall limit Need to do some more debugging in AMM7

Resubmit:


Hit wall time. No output. XML only sought 3d sbmooring which is enabled in .F90

Resubmit with one simple output to check F90 (2 moorings in one file) and no 3d sbmoorings:


STILL HITTING WALL TIME. SOMETHING WRONG WITH 3D harmonics not CORRECT (nit000_han andnitend_han). UPDATE run_nemo (using ../../XIOS_AMM60_nemo_harmIT/EXP_harmIT/run_nemo) and RESUB-MIT:


This is running and outputting 1d mooring files Run to competion (5mins for 1 day) but also creates a core dump.Data in 1d mooring file is not OK. Time axis runs L=1:0. THIS MAY WELL BE RESOLVED WITH A LONGERRUN.

NEXT STEP TO TRY AGAIN WITH 3D moorings Trim run_counter.txt Edit iodef.xml to include 3d sbmoorings:

<file id="file51" name_suffix="SB" description="Shelf break moorings"><field_group group_ref="sbmooring"/>

</file>



Iron out bug with 3D harmonics expecting extra harmonic outputs that don’t exist. Edit namelist_cfg. TURN OFF 3Dharmonics:

!-----------------------------------------------------------------------&nam_diaharm ! Harmonic analysis of tidal constituents ('key_diaharm')!-----------------------------------------------------------------------

nit000_han = 1 ! First time step used for harmonic analysisnitend_han = 75 ! Last time step used for harmonic analysisnstep_han = 15 ! Time step frequency for harmonic analysistname(1) = 'O1' ! name of constituenttname(2) = 'P1'tname(3) = 'K1'tname(4) = 'N2'tname(5) = 'M2'tname(6) = 'S2'tname(7) = 'K2'tname(8) = 'Q1'tname(9) = 'M4'

/!-----------------------------------------------------------------------&nam_dia25h ! Output 25 hour mean diagnostics!-----------------------------------------------------------------------ln_dia25h = .false.

Resubmit:


cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings

WORKED AND COMPLETED IN 5mins for 1 day 1d mooring data was outputted (though data was missing forhours 1 : 10, present from 11 : 24 ?)

This was from an executable copied from XIOS_AMM60_nemo_harmIT. Compile fresh and rerun. It makes littlesense to be developing to code bases, with and without 3d harmonics when namelist switches can be used. Make abackup copy of MY_SRC and then copy from XIOS_AMM60_nemo_harmIT:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemocp -r MY_SRC MY_SRC_1Nov16

cp -r ../XIOS_AMM60_nemo_harmIT/MY_SRC MY_SRC

Note there was an oddity in diawri.F90 with

CALL iom_put( "rhop_surf", rhop(:,:,1) ) ! surface potential density.But rhop_surf does not exist. Comment out this line.

. . .

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG




Add key_diaharm flag:

vi XIOS_AMM60_nemo/cpp_XIOS_AMM60_nemo.fcmbld::tool::fppkeys key_ldfslp key_iomput key_mpp_mpi key_netcdf4 key_tide key_bdy→˓key_jdha_init key_dynspg_ts key_vvl key_zdfgls key_dynldf_smag key_traldf_smag key_→˓traldf_c3d key_dynldf_c3d key_diaharm

Compile:



Copy executable (or check that the symbolic link is correct):: cp XIOS_AMM60_nemo/BLD/bin/nemo.exeXIOS_AMM60_nemo/EXP_SBmoorings/.



Resubmit:


With new sbmask and karen’s code. This works. Note the the sbmask is only defined on each process at the moment.

Size of 3x3 output, on each processor, which is 5236 moorings, hourly output for one day is 45G. This would needoffline compression.

Edit compression script, then submit:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmooringsvi nc_compress.pbsqsub nc_compress.pbs

This hit the Wall time limit (60mins) and so did not work. Extend wall time to 2 hours:

qsub nc_compress.pbs

This took 2hr 6mins to process one 45Gb file (1 day of data), which took 6 mins to generate..Compressed size 294Mb

Investigate better compression using on-line code. Have a trawl of James’ filespace for examples ofxmlio_server.def. Copy and update the nn_chunks_k. Note AMM60 has 51 levels, NNA has 75:

cp /work/n01/n01/jdha/ST/trunk/NEMOGCM/CONFIG/AMM7/EXP00/xmlio_server.def /work/n01/→˓n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_→˓nemo/EXP_SBmoorings/xmlio_server.defvi xmlio_server.def...!-----------------------------------------------------------------------&namnc4 ! netcdf4 chunking and compression settings

! (benign if "key_netcdf4" is not used)





!-----------------------------------------------------------------------nn_nchunks_i = 4 ! number of chunks in i-dimensionnn_nchunks_j = 4 ! number of chunks in j-dimensionnn_nchunks_k = 6 ! number of chunks in k-dimension

! setting nn_nchunks_k = jpk will give a chunk size of→˓1 in the vertical which

! is optimal for postprocessing which works→˓exclusively with horizontal slabs

ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression! (F) ignore chunking information and produce netcdf3-

→˓compatible files



Resubmit:


** Completed in 5min 57** AMM60_1h_20120601_20120601_SB.nc is still 45Gb, thoughAMM60_1h_20120601_20120601_SB001_grid_T.nc is probably smaller.


CHAPTER 4

Outputting moorings as a masked 3D array

Source: /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings

• Need to create a globally accessible mask file sbmask

• Need to write output

4.1 Create a mask file

Make some edits to move sbmask definition to dom_oce.F90:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/NEMO/→˓OPA_SRC

cp DOM/dommsk.F90 ../../CONFIG/XIOS_AMM60_nemo/MY_SRC/dommsk.F90cp DOM/dom_oce.F90 ../../CONFIG/XIOS_AMM60_nemo/MY_SRC/dom_oce.F90

Edit sbmask files:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/MY_SRCvi dommsk.F90...

vi dom_oce.F90...

Compile:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG

module add cray-hdf5-parallelmodule load cray-netcdf-hdf5parallel


41



module swap PrgEnv-cray PrgEnv-intel




Resubmit:


PENDING (1 Nov 2016) cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings

• Does the output work? YES

• How fast / slow is it? 6mins / day

• How large is the output? 45Gb / day

• Implement proper IF statements in diawri.F90

Edit domain_def.xml to output region of interest only:: vi domain_def.xml

<!– Shelf Break virtual mooring mask –> <domain id=”SB” zoom_ibegin=”0307” zoom_jbegin=”0252”zoom_ni=”0666” zoom_nj=”0986” />

Add subdomain SB to iodef.xml file:

vi iodef.xml...


→˓files -->

<file id="file51" name_suffix="_SB" description="Shelf break moorings"><field_group group_ref="sbmooring" domain_ref="SB"/>

</file>

</file_group>

Run for 1 day. Trim run_counter.txt:


4.2 Simulation plan for June 2012

Presently from run_counter.txt 6 minutes a day. Therefore 3 day chunks will fit on the 20mins queue

1264321 1265760 - 1 day

1264321 1268640 - 3 days

For June 2012 need 10 loops of 3 days:

42 Chapter 4. Outputting moorings as a masked 3D array


vi run_counter.txt1 1 7200 201001051264321 1268640

vi run_nemoexport nrestart_max=11 #31 (For one submission this number must equal the number of→˓lines in run_counter.txt)

I made these edits after the job was submitted but was still in the queue. Perhaps it will work. . .

Resubmit:


Editted finish_nemo.sh to launch compression of data:

vi finish_nemo.sh...# Compress outputqsub $JOBDIR/nc_compress.pbs

The compression script moves compressed files to OUTPUT/tmp. This needs to be cleaned when the compressed filesare checked. ** This didn’t seem to work properly:, though output seemed to compress fine the log files were lostand the job ran until walltime limit

PENDING (2 Nov 2016) cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings

• Does the output work? YES

• How fast / slow is it? > 6mins / day ? Between 5mins 14 and 7mins 48

• How large is the output? < 45Gb / day 18Gb uncompressed, 322Mb compressed

• Clean the OUTPUT/tmp directory if output is OK.

• nc_compress.pbs: 1hr 35

ACTION: copy to SAN:: rsync -uartv [email protected]:/work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings/OUTPUT/cmp*/projectsa/FASTNEt/jelt/AMM60/RUNS/2010_2013/SB_moorings/. chmod a+rx /pro-jectsa/FASTNEt/jelt/AMM60/RUNS/2010_2013/SB_moorings/cmp*nc

Job has finished but the 1 June file is missing Edit run_counter.txt to just do one day:


Here is the completed run_counter.txt file with 1 day increments. 1 1 7200 20100105 2 12643211265760 3 1265761 1267200 1265760=20120601 4 1267201 1268640 1267200=20120602 5 1268641 12700801268640=20120603 6 1270081 1271520 1270080=20120604 7 1271521 1272960 1271520=20120605 8 12729611274400 1272960=20120606 9 1274401 1275840 1274400=20120607 10 1275841 1277280 1275840=2012060811 1277281 1278720 1277280=20120609 12 1278721 1280160 1278720=20120610 13 1280161 12816001280160=20120611 14 1281601 1283040 1281600=20120612 15 1283041 1284480 1283040=20120613 16 12844811285920 1284480=20120614 17 1285921 1287360 1285920=20120615 18 1287361 1288800 1287360=2012061619 1288801 1290240 1288800=20120617 20 1290241 1291680 1290240=20120618 21 1291681 12931201291680=20120619 22 1293121 1294560 1293120=20120620 23 1294561 1296000 1294560=20120621 24 12960011297440 1296000=20120622 25 1297441 1298880 1297440=20120623 26 1298881 1300320 1298880=2012062427 1300321 1301760 1300320=20120625 28 1301761 1303200 1301760=20120626 29 1303201 1304640

4.2. Simulation plan for June 2012 43

mailto:[email protected]


1303200=20120627 30 1304641 1306080 1304640=20120628 31 1306081 1307520 1306080=20120629 32 13075211308960 1307520=20120630

ACTIONS: * Check qstat progress on 1st June. DONE. * Check progress of nc_compress. How long did ittake? 1hr 35 DONE. * Submit lots of nc_compress jobs with revised wall time estimates. DONE * Submit lastnc_compress1_9_10.pbs when only 3 files left in OUTPUT. DONE * LATER: scp cmp*nc files to SAN. DONE *gzip OUTPUT/cmp*nc files DONE * delete EXP_SBmoorings2 EXP_SBmoorings3 * move EXP_SBmoorings to/nerc

(31 Jan 2016) Over wrote run_nemo and submit_nemo.pbs by mistake. Replaced them with old copies. I think thatthey are OK but haven’t tested. Noted that extra processes are used for the XIOS steps (XIOSproc=60 instead of 40running on 92 nodes). These are probably not needed with the 3D masked array output method.

44 Chapter 4. Outputting moorings as a masked 3D array

CHAPTER 5

Compiling AMM60

Aim:

• To recompile same code as Karen used: /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP

• Use James’ XIOS file.

Load modules:


jelt@eslogin004:~> module listCurrently Loaded Modulefiles:

1) modules/3.2.10.2 17) cray-hdf5-parallel/1.8.132) eswrap/1.3.2-1.020200.1274.0 18) intel/15.0.2.1643) switch/1.0-1.0502.57058.1.58.ari 19) cray-libsci/13.2.04) craype-network-aries 20) udreg/2.3.2-1.0502.9889.2.20.ari5) craype/2.4.2 21) ugni/6.0-1.0502.10245.9.9.ari6) pbs/12.2.401.141761 22) pmi/5.0.7-1.0000.10678.155.25.ari7) craype-ivybridge 23) dmapp/7.0.1-1.0502.10246.8.47.ari8) cray-mpich/7.2.6 24) gni-headers/4.0-1.0502.10317.9.2.ari9) packages-archer 25) xpmem/0.1-2.0502.57015.1.15.ari

10) bolt/0.6 26) dvs/2.5_0.9.0-1.0502.1958.2.55.ari11) nano/2.2.6 27) alps/5.2.3-2.0502.9295.14.14.ari12) leave_time/1.0.0 28) rca/1.0.0-2.0502.57212.2.56.ari13) quickstart/1.0 29) atp/1.8.314) ack/2.14 30) PrgEnv-intel/5.2.5615) xalt/0.6.0 31) cray-netcdf-hdf5parallel/4.3.3.116) epcc-tools/6.0 32) cray-hdf5-parallel/1.8.14

Copy/synchronise with Karen’s version of code (note if this is a big copy job it can be run using qsub ~/scripts/rsync.pbs):

45


rsync -uartvL --delete /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/ARCH/ /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/ARCHrsync -uartvL --delete /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/NEMO/ /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/NEMOrsync -uartvL --delete /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/EXTERNAL/ /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_→˓BDY_VERT_INTERP/NEMOGCM/EXTERNALrsync -uartvL --delete /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/SETTE/ /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/SETTErsync -uartvL --delete /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/TOOLS/ /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/TOOLSrsync -uartvL --delete /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/fcm-make/ /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_→˓BDY_VERT_INTERP/NEMOGCM/fcm-make

Get James’ ARCH file:

cp /work/n01/n01/jdha/ST/trunk/NEMOGCM/ARCH/arch-XC_ARCHER_INTEL.fcm /work/n01/n01/→˓jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/ARCH/arch-XC_ARCHER_→˓INTEL.fcm

Make the config directory:

mkdir /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/→˓CONFIGcd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG

Copy in the makefile:

cp /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/→˓CONFIG/makenemo /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/.

Make the configuration directory (with default options which generates a cfg.txt file with options:XIOS_AMM60_nemo OPA_SRC LIM_SRC_2 NST_SRC):


Copy MY_SRC code from Karen:

rsync -uartvL --delete /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/CONFIG/AMM60smago/MY_SRC/ /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/→˓dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/MY_SRC

Copy in the compiler flags file:

cp /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/→˓CONFIG/AMM60smago/cpp_AMM60smago.fcm /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_→˓NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/cpp_XIOS_AMM60_nemo.fcm

Compile with XC_ARCHER_INTEL:

46 Chapter 5. Compiling AMM60



Make a copy of this before getting all the boundary, grid and forcing files etc:

cp -r XIOS_AMM60_nemo/ XIOS_AMM60_nemo_freshcompile

The following is done automatically in the run_nemo script:

Copy executable:

cp XIOS_AMM60_nemo/BLD/bin/nemo.exe XIOS_AMM60_nemo/EXP00/.

Link XIOS executable to EXPeriment directory:

ln -s /work/n01/n01/jdha/ST/xios-1.0/bin/xios_server.exe /work/n01/n01/jelt/NEMO/→˓NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP00/→˓xios_server.exe

21 Sept 2016 Recompile with debugging tools. Add -g -traceback

Load modules:


cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/ARCHvi arch-XC_ARCHER_INTEL.fcm

%FCFLAGS -integer-size 32 -real-size 64 -O3 -fp-model source -zero -fpp -→˓warn all -g -traceback%FFLAGS -integer-size 32 -real-size 64 -O3 -fp-model source -zero -fpp -→˓warn all -g -traceback


cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG./makenemo -n XIOS_AMM60_nemo_debug -m XC_ARCHER_INTEL -j 10


rsync -uartvL --delete /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/CONFIG/AMM60smago/MY_SRC/ /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/→˓dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_debug/MY_SRC


cp /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/→˓CONFIG/AMM60smago/cpp_AMM60smago.fcm /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_→˓NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_debug/cpp_XIOS_AMM60_nemo_debug.→˓fcm


47


./makenemo -n XIOS_AMM60_nemo_debug -m XC_ARCHER_INTEL -j 10

THIS WORKED.

Remove -g -traceback flags from arch-XC_ARCHER_INTEL.fcm:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/ARCHvi arch-XC_ARCHER_INTEL.fcm%FCFLAGS -integer-size 32 -real-size 64 -O3 -fp-model source -zero -fpp -→˓warn all%FFLAGS -integer-size 32 -real-size 64 -O3 -fp-model source -zero -fpp -→˓warn all

48 Chapter 5. Compiling AMM60

CHAPTER 6

3D Harmonic Analysis

PATH:

/work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo_harmIT

Method:

• First follow Compiling AMM60. 15 Sept 2016

• Test code on NSea simulations.Testing a new compilation with NSea experiment. New build ssh looks like oldbuild ssh. Check output: AMM60_read_plot.ipynb Looks OK. 16 Sept 2016

• Make a clean EXPeriment directory EXP_harmIT. In clean build configuration. 20 Sept 2016 /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmIT/EXP_harmIT

load modules:



cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG./makenemo -n XIOS_AMM60_nemo_harmIT -m XC_ARCHER_INTEL -j 10


rsync -uartvL --delete /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/CONFIG/AMM60smago/MY_SRC/ /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/→˓dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmIT/MY_SRC

49



Add MY_SRC code from Maria:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo_harmIT/MY_SRCmv diaharm2.F90 diaharm_.F90_kariho40#mv dia25h.F90 dia25h.F90_kariho40cp /work/n01/n01/mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC/DIA/diaharm.F90 diaharm.F90_mane1

Maria suggested I need the following but it breaks the compilation so for now comment it out. #cp /work/n01/n01/mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC/DIA/dia25h.F90 .

Create a new diaharm.F90 that merges Karen and Maria’s. Start with Maria’s:

cp diaharm.F90_mane1 diaharm.F90sdiff diaharm.F90_mane1 diaharm2.F90_kariho40

mane1 kariho40bold is the new / accepted version

INTEGER :: jh, nhan, jl | INTEGER :: jh, nhan, jk, ji

DO jh=1,jpmax_harmo | DO jk=1,jpmax_harmoDO jl=1,jpmax_harmo | DO ji=1,jpmax_harmo

IF(TRIM(tname(jh)) == Wave(jl)%cname_tide) THEN |→˓IF(TRIM(tname(jk)) == Wave(ji)%cname_tide) THEN

nb_ana=nb_ana+1 nb_ana=nb_ana+1ENDIF ENDIF

DO jh=1,nb_ana | DO jk=1,nb_anaDO jl=1,jpmax_harmo | DO ji=1,jpmax_harmo

IF (TRIM(tname(jh)) .eq. Wave(jl)%cname_tide) THEN | IF→˓(TRIM(tname(jk)) .eq. Wave(ji)%cname_tide) THEN

name(jh) = jl | name(jk) = jiEXIT EXIT

END IF END IF

! Initialize temporary arrays: ! Initialize temporary arrays:! ---------------------------- ! ----------------------------ALLOCATE( ana_temp(jpi,jpj,2*nb_ana,5,jpk) ) | ALLOCATE( ana_

→˓temp(jpi,jpj,2*nb_ana,3) )ana_temp(:,:,:,:,:) = 0._wp | ana_temp(:,:,:,:) = 0._wp

INTEGER :: ji, jj, jh, jc, nhc ,jk | INTEGER :: ji, jj, jh, jc,→˓nhc

! ssh, ub, vb are stored at the last level of 5d array |

...3D stuff only here that we keepSUBROUTINE dia_harm_end

INTEGER :: ji, jj, jh, jc, jn, nhan, jl ,jk | INTEGER :: ji, jj, jh,→˓ jc, jn, nhan, jl...

ALLOCATE( out_eta(jpi,jpj,jpk,2*nb_ana), & <& out_u (jpi,jpj,jpk,2*nb_ana), & <& out_v (jpi,jpj,jpk,2*nb_ana), & <& out_w (jpi,jpj,jpk,2*nb_ana), & <

> ALLOCATE( out_eta(jpi,jpj,2*nb_ana), &> & out_u (jpi,jpj,2*nb_ana), &


50 Chapter 6. 3D Harmonic Analysis



> & out_v (jpi,jpj,2*nb_ana) )

& out_dzi(jpi,jpj,jpk,2*nb_ana) ) <! density and Elevation: | ! Elevation:

DO jk=1,jpk <ztmp4(kun)=ana_temp(ji,jj,kun,1,jk) |

→˓ztmp4(kun)=ana_temp(ji,jj,kun,1)

out_eta(ji,jj,jk,jh ) = X1 * tmask_i(ji, | out_→˓eta(ji,jj,jh ) = X1 * tmask_i(ji,jj)

out_eta(ji,jj,jk,jh+nb_ana) = X2 * tmask_i(ji, | out_→˓eta(ji,jj,jh+nb_ana) = X2 * tmask_i(ji,jj)

! u-component of velocity | ! ubar:

On reflection the bits kept from Karen’s version are not helpful. They just seem to be a straight label swap and doesnot achieve anything.


cp /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/→˓CONFIG/AMM60smago/cpp_AMM60smago.fcm /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_→˓NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmIT/cpp_XIOS_AMM60_nemo_→˓harmIT.fcm

Check /add the key_diaharm:

vi cpp_XIOS_AMM60_nemo_harmIT.fcmbld::tool::fppkeys key_ldfslp key_iomput key_mpp_mpi key_netcdf4 key_tide key_bdy→˓key_jdha_init key_dynspg_ts key_vvl key_zdfgls key_dynldf_smag key_traldf_smag key_→˓traldf_c3d key_dynldf_c3d key_diaharm


cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG./makenemo -n XIOS_AMM60_nemo_harmIT -m XC_ARCHER_INTEL -j 10 clean./makenemo -n XIOS_AMM60_nemo_harmIT -m XC_ARCHER_INTEL -j 10

WORKED! IT COMPILES

INSERT 3 Oct:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo_harmIT/MY_SRCcp /work/n01/n01/mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC/DIA/dia25h.F90 ../MY_SRC/.cp /work/n01/n01/mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC/DIA/diainsitutem.F90 ../MY_SRC/.

Recompile:


Logs:

51


.../work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo_harmIT/BLD/ppsrc/nemo/dia25h.f90(24): error #6580: Name in only-→˓list does not exist. [EN]

USE zdf_oce, ONLY: en----------------------^...Build failed on Mon Oct 3 15:56:07 2016.

Build failed in dia25h.F90...

USE zdf_oce, ONLY: en...

Compare zdf_oce.F90:

diff /work/n01/n01/mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC/ZDF/zdf_oce.F90 /work/n01/n01/→˓jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/NEMO/OPA_SRC/ZDF/zdf_→˓oce.F9045,47d44< REAL(wp), PUBLIC, ALLOCATABLE, SAVE, DIMENSION(:,:,:) :: avt_k , avm_k ! not→˓enhanced Kz< REAL(wp), PUBLIC, ALLOCATABLE, SAVE, DIMENSION(:,:,:) :: avmu_k, avmv_k ! not→˓enhanced Kz< REAL(wp), PUBLIC, ALLOCATABLE, SAVE, DIMENSION(:,:,:) :: en !:→˓now turbulent kinetic energy [m2/s2]51c48< !! $Id: zdf_oce.F90 6204 2016-01-04 13:47:06Z cetlod $---> !! $Id: zdf_oce.F90 5038 2015-01-20 14:26:13Z jamesharle $65,68c62< & avmv (jpi,jpj,jpk), avt (jpi,jpj,jpk) , &< & avt_k (jpi,jpj,jpk), avm_k (jpi,jpj,jpk) , &< & avmu_k(jpi,jpj,jpk), avmv_k(jpi,jpj,jpk) , &< & en (jpi,jpj,jpk), STAT = zdf_oce_alloc )---> & avmv(jpi,jpj,jpk), avt(jpi,jpj,jpk) , STAT = zdf_oce_alloc→˓)

I.e. Maria has a newer version of zdf_oce.F90. Copy it:

cp /work/n01/n01/mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC/ZDF/zdf_oce.F90 /work/n01/n01/→˓jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_→˓nemo_harmIT/MY_SRC/zdf_oce.F90

Also there are knock ons in zdfgls.F90. Copy it (accidentally overwrote the OPA_SRC/ZDF/ version):

# cp /work/n01/n01/mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC/ZDF/zdfgls.F90 /work/n01/n01/→˓jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/NEMO/OPA_SRC/ZDF/→˓zdfgls.F90

cp /work/n01/n01/mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC/ZDF/zdfgls.F90 /work/n01/n01/→˓jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_→˓nemo_harmIT/MY_SRC/zdfgls.F90

Recompile:




IT WORKED!

6.1 Make a clean EXPeriment: EXP_harmIT

Note that this EXP requires a modified version of field_def.xml. Previously we used Karen’s reference version.This requires a modification to run_nemo

Move to configuration directory and create EXPeriment directory:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo_harmITmkdir EXP_harmIT

Copy run script into job directory:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/run_nemo EXP_harmIT/run_nemocp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/submit_nemo.pbs EXP_harmIT/submit_nemo.pbs

Make a SHARED directory, and copy field_def.xml to it:

mkdir /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/→˓CONFIG/SHAREDcp /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/→˓CONFIG/SHARED/field_def.xml /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_→˓BDY_VERT_INTERP/NEMOGCM/CONFIG/SHARED/.

Edit run script (Note I only want one restart at this stage):

vi EXP_harmIT/run_nemoexport RUNNAME=EXP_harmITexport YEARrun='2012'export HOMEDIR=/work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmITexportnrestart_max=2#31 (For one submission this number must equal the number of→˓lines in run_counter.txt)Edit the SHAREDDIR path to be mine and not Karen'sexport SHAREDDIR=$HOMEDIR/../SHARED # Config directory

# edit where the namelist_cfg is updated to include harmonic analysis...273c\nit000_han = "$nn_it000"274c\nitend_han = "$nitend" " $JOBDIR/namelist_cfg > $WDIR/namelist_cfg

Edit submit script. Extended wall time to see how long it takes:

vi submit_nemo.pbs#PBS -N AMM60_harmIT#PBS -l walltime=00:20:00

6.1. Make a clean EXPeriment: EXP_harmIT 53

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/iodef.xml EXP_harmIT/iodef.xml

Add in tidal diagnostics (copied from /work/n01/n01/mane1/AMM7_w/iodef.xml). Comment out most, justto get it working:

vi EXP_harmIT/iodef.xml<file_group id="1d" output_freq="1d" output_level="10" enabled=".TRUE." > ...

<file id="file8" name_suffix="_Tides" description="tidal harmonics" ><field field_ref="e3t" /><field field_ref="gdept"/><field field_ref="M2x_ro" name="M2x_ro" long_name="M2 ro real part"

→˓ /><field field_ref="M2y_ro" name="M2y_ro" long_name="M2 ro imaaginary

→˓part" /><field field_ref="M2x_u" name="M2x_u" long_name="M2 current bcl-

→˓baro i-axis harmonic real " /><field field_ref="M2y_u" name="M2y_u" long_name="M2 current bcl-

→˓baro i-axis harmonic imaginary " /><field field_ref="M2x_v" name="M2x_v" long_name="M2 current bcl-

→˓baro j-axis harmonic real " /><field field_ref="M2y_v" name="M2y_v" long_name="M2 current bcl-

→˓baro j-axis harmonic imaginary " /><field field_ref="M2x_w" name="M2x_w" long_name="M2 current

→˓vertical harmonic real part " /><field field_ref="M2y_w" name="M2y_w" long_name="M2 current

→˓vertical imaginary part " /><field field_ref="M2x_SSH" name="M2x_SSH" long_name="M2 ro real part"

→˓ /><field field_ref="M2y_SSH" name="M2y_SSH" long_name="M2 ro imaaginary

→˓part" /><field field_ref="M2x_u2d" name="M2x_u2d" long_name="M2 current baro i-

→˓axis harmonic real " /><field field_ref="M2y_u2d" name="M2y_u2d" long_name="M2 current baro i-

→˓axis harmonic imaginary " /><field field_ref="M2x_v2d" name="M2x_v2d" long_name="M2 current baro j-

→˓axis harmonic real " /><field field_ref="M2y_v2d" name="M2y_v2d" long_name="M2 current baro j-

→˓axis harmonic imaginary " /><field field_ref="M2x_dzi" name="M2x_dzi" long_name="M2 isopycnal

→˓elevations harmonic real part " /><field field_ref="M2y_dzi" name="M2y_dzi" long_name="M2 isopycnal

→˓elevations imaginary part " /><field field_ref="M2x_tabx" name="M2x_tabx" long_name="M2 bottom shear

→˓stress i-axis real " /><field field_ref="M2y_tabx" name="M2y_tabx" long_name="M2 bottom shear

→˓stress i-axes imaginary " /><field field_ref="M2x_taby" name="M2x_taby" long_name="M2 bottom shear

→˓stress j-axes real " /><field field_ref="M2y_taby" name="M2y_taby" long_name="M2 bottom shear

→˓stress j-axis imaginary " /><!--

<field field_ref="S2x_ro" name="S2x_ro" long_name="S2 ro real part"→˓ /> (continues on next page)

<field field_ref="S2y_ro" name="S2y_ro" long_name="S2 ro imaaginary→˓part" />

<field field_ref="S2x_u" name="S2x_u" long_name="S2 current bcl-→˓baro i-axis harmonic real " />

<field field_ref="S2y_u" name="S2y_u" long_name="S2 current bcl-→˓baro i-axis harmonic imaginary " />

<field field_ref="S2x_v" name="S2x_v" long_name="S2 current bcl-→˓baro j-axis harmonic real " />

<field field_ref="S2y_v" name="S2y_v" long_name="S2 current bcl-→˓baro j-axis harmonic imaginary " />

<field field_ref="S2x_w" name="S2x_w" long_name="S2 current→˓vertical harmonic real part " />

<field field_ref="S2y_w" name="S2y_w" long_name="S2 current→˓vertical imaginary part " />

<field field_ref="S2x_SSH" name="S2x_SSH" long_name="S2 ro real part"→˓ />

<field field_ref="S2y_SSH" name="S2y_SSH" long_name="S2 ro imaaginary→˓part" />

<field field_ref="S2x_u2d" name="S2x_u2d" long_name="S2 current baro i-→˓axis harmonic real " />

<field field_ref="S2y_u2d" name="S2y_u2d" long_name="S2 current baro i-→˓axis harmonic imaginary " />

<field field_ref="S2x_v2d" name="S2x_v2d" long_name="S2 current baro j-→˓axis harmonic real " />

<field field_ref="S2y_v2d" name="S2y_v2d" long_name="S2 current baro j-→˓axis harmonic imaginary " />

<field field_ref="S2x_dzi" name="S2x_dzi" long_name="S2 isopycnal→˓elevations harmonic real part " />

<field field_ref="S2y_dzi" name="S2y_dzi" long_name="S2 isopycnal→˓elevations imaginary part " />

<field field_ref="S2x_tabx" name="S2x_tabx" long_name="S2 bottom shear→˓stress i-axis real " />

<field field_ref="S2y_tabx" name="S2y_tabx" long_name="S2 bottom shear→˓stress i-axes imaginary " />

<field field_ref="S2x_taby" name="S2x_taby" long_name="S2 bottom shear→˓stress j-axes real " />

<field field_ref="S2y_taby" name="S2y_taby" long_name="S2 bottom shear→˓stress j-axis imaginary " />-->

</file>...

Edit field_def.xml in SHARED directory. Copy and paste from /work/n01/n01/mane1/AMM7_w/field_def.xml:

vi /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓SHARED/field_def.xml

<field_group id="Tides_T" grid_ref="grid_T_2D" operation="once" ><field id="M2x_ro" long_name="M2 ro/roa harmonic real part " unit=

→˓"none" grid_ref="grid_T_3D" /><field id="M2y_ro" long_name="M2 ro/roa harmonic imaginary part"

→˓unit="none" grid_ref="grid_T_3D" /><field id="S2x_ro" long_name="S2 ro/roa harmonic real part " unit=

→˓"none" grid_ref="grid_T_3D" />



<field id="Mfx_dzi" long_name="Mf isopycnal elevation harmonic real→˓part " unit="m" />

<field id="Mfy_dzi" long_name="Mf isopycnal elevation harmonic→˓imaginary part " unit="m" />

<field id="Mmx_dzi" long_name="Mm isopycnal elevation harmonic real→˓part " unit="m" />

<field id="Mmy_dzi" long_name="Mm isopycnal elevation harmonic→˓imaginary part " unit="m" />

<field id="S1x_dzi" long_name="S1 isopycnal elevation harmonic real→˓part " unit="m" />

<field id="S1y_dzi" long_name="S1 isopycnal elevation harmonic→˓imaginary part " unit="m" />

</field_group>


cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/domain_def.xml EXP_harmIT/domain_def.xml

Copy finish_nemo.sh into job directory:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/finish_nemo.sh EXP_harmIT/finish_nemo.sh

Link restart files:

mkdir EXP_harmIT/RESTARTln -s /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/→˓NEMOGCM/CONFIG/AMM60smago/EXPD376/RESTART/01264320 EXP_harmIT/RESTART/.

Create run_counter.txt into job directory. Start from 23:30 31 May 2012 (I think). Run for 5 days:

vi EXP_harmIT/run_counter.txt1 1 7200 201001052 1264321 1271520

Edit run_counter, to run for two days (2*1440 = 2880 minutes):


Copy in namelists:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/namelist_ref EXP_harmIT/.cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/namelist_cfg EXP_harmIT/.



6.2 Try Two Simultaneous experiments:

6.3 First Run

Submit run (5min wall time). Fiddled with iodef.xml to output HOURLY:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo_harmIT/EXP_harmIT./run_nemo3972846.sdb


Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----3972846.sdb jelt standard AMM60_harm -- 92 220 -- 00:05 Q -- <--→˓Does it WORK?OUTPUT SHOULD BE 3D harmonics, outputted HOURLY over 2 days

Broke.Try cleaning up iodef.xml (remove comments):cp iodef.xml ifdef.xml_tmp

Several bugs. Resubmit

Trim run_counter.txt. Resubmit:



Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----3977937.sdb jelt standard AMM60_harm -- 92 220 -- 00:05 Q -- <--→˓DOES THIS WORK.PREVIOUSLY ALWAYS GOT CORE DUMPS.OUTPUT SHOULD BE 3D harmonics, outputted hourly over 2 days (7 Oct 2016)

Hmm something went wrong:

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo_harmIT/EXP_harmIT

Something went wrong in field_def.xml.Edit the comment tags to clean up the closure of comments. copy edits back toEXP_harmIT> mv WDIR/field_def.xml ../../SHARED/field_def.xml

Trim run_counter.txt and resubmit:



3982223.sdb


Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----3982223.sdb jelt standard AMM60_harm -- 92 220 -- 00:05 Q --<--→˓DOES THIS WORK.PREVIOUSLY ALWAYS GOT CORE DUMPS.OUTPUT SHOULD BE 3D harmonics, outputted hourly over 2 days (10 Oct 2016)

This WORKED. Resubmit full 3D harmonics list in iodef.xml on a 20minute job.

BROKE. NO OUTPUT. RAN TO WALLTIME.

Check output log:

less ocean.output...dia_harm_init: Tidal harmonic analysis initialization~~~~~~~First time step used for analysis: nit000_han= 1Last time step used for analysis: nitend_han= 75Time step frequency for harmonic analysis: nstep_han= 15

===>>> : E R R O R===========

dia_harm_init : nit000_han must be greater than nit000

This may well have broken the model.Fix and resubmit on a 5 minute job on a two day simulation.

Edit run_counter.txt to run for two days:


Edit namelist_cfg harmonic analysis variables:

&nam_diaharm ! Harmonic analysis of tidal constituents ('key_diaharm')!-----------------------------------------------------------------------

nit000_han = 1 ! First time step used for harmonic analysisnitend_han = 75 ! Last time step used for harmonic analysisnstep_han = 15 ! Time step frequency for harmonic analysis

Change into:

&nam_diaharm ! Harmonic analysis of tidal constituents ('key_diaharm')!-----------------------------------------------------------------------

nit000_han = 1264321 ! First time step used for harmonic analysisnitend_han = 1271520 ! Last time step used for harmonic analysisnstep_han = 15 ! Time step frequency for harmonic analysis

6.3. First Run 59


This needs to be added to the run_nemo script

Edit wall time:

vi submit_nemo.pbs..#PBS -l walltime=00:05:00..

Submit:



Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----3984287.sdb jelt standard AMM60_harm -- 92 220 -- 00:05 Q -- <--→˓DOES THIS WORK?EXPECT TWO DAYS OF 3D HARMONIC OUTPUT (11 Oct 2016)

YES. But it hit the 5 min walltime limit. Though the harmonic output was started.

less time.step1266175

This is 30.9 hours after the start. Hence 5mins wall-time –> 1.5days simulation. Hence 31days simulation requires103 mins wall time.

First try and complete the two day simulation with 10mins walltime. First try and complete the five day simulationwith 20mins walltime.

Edit walltime:


Edit run_counter.txt:

vi run_counter.txt1 1 7200 201001052 1264321 1271520 # THIS IS A 5 DAY SIMULATION!!

Submit:



Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----3985531.sdb jelt standard AMM60_harm -- 92 220 -- 00:20 Q --

EXPECT hourly 3D harmonics from 5 day simulation (11 Oct 2016)

cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmIT/EXP_harmIT



Check output:

ncdump -h WDIR/AMM60_1h_20120601_20120605_Tides.nc

Previously This looks fine with 72 hourly outputs.Now resubmitted with a 20min wall time.

12 Oct 2016

run_counter.txt ready for next run.OUTPUT written, 65Gb file: AMM60_1h_20120601_20120605_Tides.nclogs seem fine.

Check output:

module load cray-netcdfncdump -h OUTPUT/AMM60_1h_20120601_20120605_Tides.nc

Stuff is there Action IS IT READABLE / PLOTTABLE?

NO!

ferret:

yes? use AMM60_1h_20120601_20120605_Tides.ncyes? sh da

currently SET data sets:1> ./AMM60_1h_20120601_20120605_Tides.nc (default)

name title I J K L M→˓ NNAV_LAT_GRID_T

Latitude 1:1120 1:1440 ... ... ...→˓ ...NAV_LON_GRID_T

Longitude 1:1120 1:1440 ... ... ...→˓ ...NAV_LAT_GRID_U

Latitude 1:1120 1:1440 ... ... ...→˓ ...NAV_LON_GRID_U

Longitude 1:1120 1:1440 ... ... ...→˓ ...NAV_LAT_GRID_V

Latitude 1:1120 1:1440 ... ... ...→˓ ...NAV_LON_GRID_V

Longitude 1:1120 1:1440 ... ... ...→˓ ...NAV_LAT_GRID_W

Latitude 1:1120 1:1440 ... ... ...→˓ ...


6.3. First Run 61



NAV_LON_GRID_WLongitude 1:1120 1:1440 ... ... ...

→˓ ...E3T T-cell thickness 1:1120 1:1440 1:51 1:0 ...→˓ ...TIME_CENTERED

Time axis ... ... ... 1:0 ...→˓ ...TIME_CENTERED_BOUNDS

1:2 ... ... 1:0 ...→˓ ...TIME_COUNTER

Time axis ... ... ... 1:0 ...→˓ ...TIME_COUNTER_BOUNDS

1:2 ... ... 1:0 ...→˓ ...GDEPT depth 1:1120 1:1440 1:51 1:0 ...→˓ ...M2X_RO M2 ro real part 1:1120 1:1440 1:51 ... ...→˓ ...M2Y_RO M2 ro imaaginary part 1:1120 1:1440 1:51 ... ...→˓ ...M2X_U M2 current bcl-baro i-axis harm 1:1120 1:1440 1:51 ... ...→˓ ...M2Y_U M2 current bcl-baro i-axis harm 1:1120 1:1440 1:51 ... ...→˓ ...M2X_V M2 current bcl-baro j-axis harm 1:1120 1:1440 1:51 ... ...→˓ ...M2Y_V M2 current bcl-baro j-axis harm 1:1120 1:1440 1:51 ... ...→˓ ...M2X_W M2 current vertical harmonic re 1:1120 1:1440 1:51 ... ...→˓ ...M2Y_W M2 current vertical imaginary p 1:1120 1:1440 1:51 ... ...→˓ ...M2X_SSH M2 ro real part 1:1120 1:1440 ... ... ...→˓ ...M2Y_SSH M2 ro imaaginary part 1:1120 1:1440 ... ... ...→˓ ...

SHADE M2X_SSH returns No valid data (19 Oct 2016)

Possible issues:

• namelist problems in copying variables from run_nemo.scr

• inappropriate writing frequencies for harmonics in XML or namelist_cfg

Compare against:: /work/n01/n01/mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00

Maria output XML at 1day freq. I used 1h

Maria: namelist_cfg:

vi namelist_cfg

!-----------------------------------------------------------------------





&namrun ! parameters of the run!-----------------------------------------------------------------------

nn_no = 0 ! job number (no more used...)cn_exp = "GA" ! experience name

**nn_it000 = 1****nn_itend = 25920**nn_date0 = 19820201nn_rstctl = 0 ! restart control = 0 nit000 is not compared to the restart file

→˓value! = 1 use ndate0 in namelist (not the value in the

→˓restart file)! = 2 calendar parameters read in the restart file

nn_leapy = 1 ! Leap year calendar (1) or not (0)ln_rstart = .true.cn_ocerst_in = "restart" ! suffix of ocean restart name (input)cn_ocerst_out = "restart" ! suffix of ocean restart name (output)cn_ocerst_indir = "./" ! directory of ocean restart name (input)cn_ocerst_outdir = "./" ! directory of ocean restart name (output)nn_istate = 0 ! output the initial state (1) or not (0)nn_stock = 25920nn_write = 25920......

!-----------------------------------------------------------------------&nam_diaharm ! Harmonic analysis of tidal constituents ('key_diaharm')!-----------------------------------------------------------------------

**nit000_han = 1** ! 105121 ! First time step used for harmonic analysis

**nitend_han = 25920** ! 105120 ! 210528 ! Last time step used for harmonic→˓analysis

Note the han terms are the same as the start and end of the simulation

Check the base file:

vi EXP_harmIT/namelist_cfg

!-----------------------------------------------------------------------&namrun ! parameters of the run!-----------------------------------------------------------------------nn_no = 0 ! job number (no more used...)

cn_exp = "AMM60" ! experience name

**nn_it000 = 1****nn_itend = 144**nn_date0 = 20100105

nn_leapy = 1 ! Leap year calendar (1) or not (0)ln_rstart = .false.

nn_rstctl = 2 ! restart control ==> activated only if ln_rstart=Tcn_ocerst_in = "restart" ! suffix of ocean restart name (input)cn_ocerst_out = "restart" ! suffix of ocean restart name (input)nn_istate = 0 ! output the initial state (1) or not (0)

nn_stock = 1440nn_write = 1440

ln_dimgnnn = .false. ! DIMG file format: 1 file for all processors (F) or by→˓processor (T)

ln_mskland = .false. ! mask land points in NetCDF outputs (costly: + ~15%)ln_clobber = .false. ! clobber (overwrite) an existing filenn_chunksz = 0 ! chunksize (bytes) for NetCDF file (works only with iom_

→˓nf90 routines) (continues on next page)

6.3. First Run 63



/...!-----------------------------------------------------------------------&nam_diaharm ! Harmonic analysis of tidal constituents ('key_diaharm')!-----------------------------------------------------------------------

**nit000_han = 1264321** ! First time step used for harmonic analysis

**nitend_han = 1271520** ! Last time step used for harmonic analysisnstep_han = 15 ! Time step frequency for harmonic analysis

Check the editted run file:

vi LOGS/01271520/namelist_cfg

!-----------------------------------------------------------------------&namrun ! parameters of the run!-----------------------------------------------------------------------nn_no = 0 ! job number (no more used...)

cn_exp = "AMM60" ! experience name

**nn_it000 = 1264321****nn_itend = 1271520**nn_date0 = 20100105

nn_leapy = 1 ! Leap year calendar (1) or not (0)ln_rstart = .true.

nn_rstctl = 2 ! restart control ==> activated only if ln_rstart=Tcn_ocerst_in = "restart" ! suffix of ocean restart name (input)cn_ocerst_out = "restart" ! suffix of ocean restart name (input)

nn_stock = 1271520nn_write = 1271520nn_write = 1440

ln_dimgnnn = .false. ! DIMG file format: 1 file for all processors (F) or by→˓processor (T)

ln_mskland = .false. ! mask land points in NetCDF outputs (costly: + ~15%)ln_clobber = .false. ! clobber (overwrite) an existing filenn_chunksz = 0 ! chunksize (bytes) for NetCDF file (works only with iom_

→˓nf90 routines)...!-----------------------------------------------------------------------&nam_diaharm ! Harmonic analysis of tidal constituents ('key_diaharm')!-----------------------------------------------------------------------

**nit000_han = 1264321****nitend_han = 1271520**

There is a problem with the output frequency of data (uneditted):

nn_istate = 0 ! output the initial state (1) or not (0)nn_stock = 1440nn_write = 1440

Because the sed writing is displaced by one line overwriting nn_istate and leaving nn_write unchanged:

nn_stock = 1271520nn_write = 1271520nn_write = 1440

Edit run_nemo sed replacement in namelist_cfg

Edit iodef.xml to output at 1day freq (like Maria):


vi iodef.xml...<file_group id="1d" output_freq="1d" output_level="10" enabled=".TRUE."> <file id="file8" name_suffix="_Tides" description="tidal harmonics" ><field field_ref="e3t" /><field field_ref="gdept"/><field field_ref="M2x_ro" name="M2x_ro" long_name="M2 ro real part"

→˓ />

Edit a 5 day simulation:


Edit submit_nemo.pbs:

vi submit_nemo.pbs...#PBS -l walltime=00:20:00

Resubmit:

run_nemo4003527.sdb



Output in WDIR with variables and axes but values all seem to be zero, or something not good. ncdump, FERRET,ncview and python didn’t workJob run until to exceeded the wall time at 20mins: run_counter.txt not changed: 2 1264321 1271520

less time.step: 1271519. This is one minute from the end!

Something went wrong:: less WDIR/stdouterr

aprun: Apid 23790262: Caught signal Terminated, sending to application forrtl: error (78): process killed(SIGTERM) Image PC Routine Line Source nemo.exe 0000000001761A91 Unknown Unknown Unknown

22 Oct 2016

If there is something going wrong because of memory resource, can I scale what Maria did to get AMM7 working?

Job terminated because of wall time limit. Edit walltime:



6.3. First Run 65


Submit:




run_counter.txt was added toOUTPUT/AMM60_1d_20120601_20120605_Tides.nc 30Gbcore dumprun for 21mins (allowed 25)less time.step: 1271519

less stdouterr:

> Error [CObjectFactory::GetObject(const StdString & id)] :In file '/work/n01/n01/jdha/ST/xios-1.0/src/object_factory_impl.hpp', line 79 -> [ id→˓= 2N2x_ro, U = field ] object is not referenced !

End of log file:

``tail -100 LOGS/01271520/ocean.output_EXP_harmIT``

...written ok

anharmo_end: kt=nitend_han: Perform harmonic analysis~~~~~~~~~~~~

dia_wri_harm : Write harmonic analysis results

So, there was a problem with writing the harmonic outputs. I really don’t understand where the 2N2x_ro problemcomes from. This variable doesn’t exist.

diaharm.F90 calls iom_put for constucted variable *_ro . . . :

CALL iom_put( TRIM(tname(jh))//'x_ro', z3real(:,:,:) )CALL iom_put( TRIM(tname(jh))//'y_ro', z3im (:,:,:) )CALL iom_put( TRIM(tname(jh))//'x_SSH', z2real(:,:) )CALL iom_put( TRIM(tname(jh))//'y_SSH', z2im (:,:) )

NEED TO EDIT THE NAMELIST TO ONLY INCLUDE A FEW HARMONICS FOR ANALYSIS. E.g Maria’snamelist_cfg:

vi /work/n01/n01/mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00/namelist_cfg!-----------------------------------------------------------------------&nam_diaharm ! Harmonic analysis of tidal constituents ('key_diaharm')!-----------------------------------------------------------------------

nit000_han = 1 ! 105121 ! First time step used for harmonic analysisnitend_han = 25920 ! 105120 ! 210528 ! Last time step used for harmonic analysisnstep_han = 12 ! Time step frequency for harmonic analysistname(1) = 'O1' ! name of constituent



tname(2) = 'P1'tname(3) = 'K1'tname(4) = 'N2'tname(5) = 'M2'tname(6) = 'S2'tname(7) = 'K2'tname(8) = 'Q1'tname(9) = 'M4'

/

Edit /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmIT/EXP_harmIT/namelist_cfg accordingly:

vi namelist_cfg!-----------------------------------------------------------------------&nam_diaharm ! Harmonic analysis of tidal constituents ('key_diaharm')!-----------------------------------------------------------------------

nit000_han = 1264321 ! First time step used for harmonic analysisnitend_han = 1271520 ! Last time step used for harmonic analysisnstep_han = 15 ! Time step frequency for harmonic analysistname(1) = 'O1' ! name of constituenttname(2) = 'P1'tname(3) = 'K1'tname(4) = 'N2'tname(5) = 'M2'tname(6) = 'S2'tname(7) = 'K2'tname(8) = 'Q1'tname(9) = 'M4'

/

Also need to add switch for 25h diagnostics (add it after the above harmonics so it does not affect sed invokation inrun_nemo):

!-----------------------------------------------------------------------&nam_dia25h ! Output 25 hour mean diagnostics!-----------------------------------------------------------------------

ln_dia25h = .true.

Also added this to namelist_ref:

vi namelist_ref!-----------------------------------------------------------------------&nam_dia25h ! Output 25 hour mean diagnostics!-----------------------------------------------------------------------

ln_dia25h = .true.

Add in 25hour diagnostics:

vi iodef.xml<file_group id="1d" output_freq="1d" output_level="10" enabled=".TRUE." > <file id="file51" name_suffix="_grid_T" description="ocean T grid variables" ><field field_ref="e3t" /><field field_ref="gdept"/><field field_ref="temper25h" name="temper25h" long_name="sea_water_potential_

→˓temperature" />(continues on next page)

6.3. First Run 67



<field field_ref="salin25h" name="salin25h" long_name="sea_water_salinity"→˓ />

<field field_ref="ssh25h" name="ssh25h" long_name="sea_surface_height_→˓above_geoid" />

<field field_ref="mldr10_1" name="mldr10_1" long_name="Mixed Layer Depth 0.01→˓ref.10m" /></file>

<file id="file53" name_suffix="_grid_U" description="ocean U grid variables" ><field field_ref="e3u" /><field field_ref="gdepu" /><field field_ref="ssu" name="uos" long_name="sea_surface_x_velocity"

→˓ /><field field_ref="uoce" name="uo" long_name="sea_water_x_velocity" /><field field_ref="vozocrtx25h" name="uo25h" long_name="sea_water_x_velocity" /><field field_ref="ubar" name="ubar" long_name="barotropic_x_velocity" /

→˓></file>

<file id="file54" name_suffix="_grid_V" description="ocean V grid variables" ><field field_ref="e3v" /><field field_ref="gdepv" /><field field_ref="ssv" name="vos" long_name="sea_surface_y_velocity"

→˓ /><field field_ref="voce" name="vo" long_name="sea_water_y_velocity" /><field field_ref="vomecrty25h" name="vo25h" long_name="sea_water_y_velocity" /><field field_ref="vbar" name="vbar" long_name="barotropic_y_velocity" /

→˓></file>

<file id="file55" name_suffix="_grid_W" description="ocean W grid variables" ><field field_ref="e3w" /><field field_ref="gdepw" /><field field_ref="vomecrtz25h" name="wo25h" long_name="ocean vertical velocity

→˓" /><field field_ref="eps25h" name="eps25h" long_name="TKE dissipation rate

→˓25h " /><field field_ref="N2_25h" name="N2_25h" long_name="25h mean Brent-Viasala

→˓" /><field field_ref="S2_25h" name="S2_25h" long_name="25h squared shear" /><field field_ref="tke25h" name="TKE25h" long_name="25h vertical kinetic

→˓energy" /></file>

<file id="file8" name_suffix="_Tides" description="tidal harmonics" ><field field_ref="e3t" /><field field_ref="gdept"/><field field_ref="M2x_ro" name="M2x_ro" long_name="M2 ro real part"

→˓ />...

ALSO ENSURE THAT THESE ARE IN FIELD_DEF.XML, IN /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/SHARED

Prepare and then resubmit:






EXPECT hourly 3D harmonics from 5 day simulation (22 Oct 2016) EXPECT 25hourly outputs too


NB without the 25h diagnostics (from a previous run):

| *30G AMM60_1d_20120601_20120605_Tides.nc*| *19 mins of simulation for 5 days*

*For some reason there is no evidence of dia25h being initialised in the log files. It is as if dia25.F90 was notcompiled. . .

*Note that the compile keys are different for AMM60 and Maria’s AMM7 runs: https://www.evernote.com/shard/s523/nl/2147483647/9efc56c7-6965-4baf-b8be-b05f4e2b8d4c/

*Recompile and resubmit

PLAN:

• Add in 25hr diagnostics. Check

• Do one, or two, month simulation

Edit run_counter, to run for two months (June and July 2012 = 87,840 mins, 61 days):

cd EXP_harmITvi run_counter.txt1 1 7200 201001052 1264321 1352160

Thread with a second simulation that was severed when I found this trunk simulation had problems

3D harmonics in AMM60 is ‘solved (I think...) in clean_3D_Harmonic_analysis<clean_3D_Harmonic_analysis.html>‘_.

6.3. First Run 69

https://www.evernote.com/shard/s523/nl/2147483647/9efc56c7-6965-4baf-b8be-b05f4e2b8d4c/

https://www.evernote.com/shard/s523/nl/2147483647/9efc56c7-6965-4baf-b8be-b05f4e2b8d4c/

clean_3D_Harmonic_analysis.index

spare_3D_Harmoninc_analyis.html

CHAPTER 7

Debugging XIOS in AMM7

Debugging XML errors in AMM60 is time consuming because of the queue duration. Here debug with AMM7 andsubmissions to the short queue. NB: I had trouble getting a new clean run working. No docs! So I’ve just startedworing in the following:

Source: cd /work/n01/n01/jelt/gmaya/NEMO/CONFIG/XIOS_AMM7_nemo2/EXP00

Test it works:

./run_nemo.sh annualrun.pbs 12 16 192 1981 1 13990406.sdb

This works!

7.1 Write multiple copies of same variable in single file

Make a safe copy of iodef.xml then edit a working copy with two different tos1 and tos2 variables:

mv iodef.xml iodef.xml_141016cp ../../XIOS_AMM7_nemo/EXP00/iodef.xml iodef.xml

Edit the working XML file

<file_group id=”1h” output_freq=”1h” output_level=”10” enabled=”.TRUE.” > <!– 1h files –>

<file id=”file51” name_suffix=”_grid_T” description=”ocean T grid variables” >

<field_group id=”test1”> <field field_ref=”sst” name=”tos1” />

</field_group> <field_group id=”test2”>

<field field_ref=”sst” name=”tos2” />

</field_group>

</file>

71

</file_group>

Resubmit:


This WORKS (18 Oct 2016). Mount ARCHER with sshfs and visualise in FERRET:

jeff% cd /Volumes/archer/gmaya/NEMO/CONFIG/XIOS_AMM7_nemo2/EXP00[livmaf:CONFIG/XIOS_AMM7_nemo2/EXP00] jeff% ferretyes? use amm7_1h_19810101_19810110_grid_T.ncyes? shade/L=240 TOS2-TOS1yes? shade/L=240 TOS1yes? shade/L=240 TOS2

7.2 Write two copies of same variable in different domains in singlefile

Edit the domain_def.xml file (these points are Malin, SB001, and North Sea, SB002):

<domain id="SB001" zoom_ibegin="0100" zoom_jbegin="0250" zoom_ni="2" zoom_nj="2" /><domain id="SB002" zoom_ibegin="0200" zoom_jbegin="0250" zoom_ni="2" zoom_nj="2" />

Edit the iodef.xml file:

<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE." > 

<file id="file51" name_suffix="_grid_T" description="ocean T grid variables" ><field_group id="test1" domain_ref="SB001"><field field_ref="sst" name="tos1" />

</field_group><field_group id="test2" domain_ref="SB002"><field field_ref="sst" name="tos2" />


</file_group>

Resubmit:


THIS WORKS! and outputs LAT and LON variables for each mooring:

yes? sh dacurrently SET data sets:

1> ./amm7_1h_19810101_19810110_grid_T.nc (default)name title I J K L M→˓ NNAV_LAT_SB001

Latitude 1:2 1:2 ... ... ...→˓ ...NAV_LON_SB001

Longitude 1:2 1:2 ... ... ...→˓ ...


72 Chapter 7. Debugging XIOS in AMM7

NAV_LAT_SB002Latitude 1:2 1:2 ... ... ...

→˓ ...NAV_LON_SB002

Longitude 1:2 1:2 ... ... ...→˓ ...TOS1 sea surface temperature 1:2 1:2 ... 1:240 ...→˓ ...TIME_CENTERED

Time axis ... ... ... 1:240 ...→˓ ...TIME_CENTERED_BOUNDS

1:2 ... ... 1:240 ...→˓ ...TOS2 sea surface temperature 1:2 1:2 ... 1:240 ...→˓ ...

Edit: if iodef.xml has long_name the same for both variables does it work?:

vi iodef.xml...<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE." > <file id="file51" name_suffix="_grid_T" description="ocean T grid variables" ><field_group id="test1" domain_ref="SB001"><field field_ref="sst" name="tos1" long_name="sea surface temperature" /></field_group><field_group id="test2" domain_ref="SB002"><field field_ref="sst" name="tos2" long_name="sea surface temperature" /></field_group>


Resubmit:


YES. This also works.

Can I break it if I copy code from the failing AMM60 iodef.xml file?:

<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE."> "<file id="file0000" name_suffix="_SB_grid_T" description="ocean T grid



→˓salinity" /><field field_ref="tpt_dep" name="depth_SB001" /><field field_ref="uoce" name="vozocrtx_SB001" long_name="sea water

→˓x velocity" />(continues on next page)

7.2. Write two copies of same variable in different domains in single file 73



<field field_ref="voce" name="vomecrty_SB001" long_name="sea water→˓y velocity" />

<field field_ref="woce" name="vovecrtz_SB001" long_name="sea water→˓w velocity" />

<field field_ref="utau" name="utau_SB001" long_name="surface→˓downward x stress" />

<field field_ref="vtau" name="vtau_SB001" long_name="surface→˓downward y stress" />

<field field_ref="ssh" name="ssh_SB001" long_name="sea→˓surface height"/>

</field_group><field_group id="1h_SB002_grid_T" domain_ref="SB002" ><field field_ref="toce" name="thetao_SB002" long_name="sea water



→˓x velocity" /><field field_ref="voce" name="vomecrty_SB002" long_name="sea water

→˓y velocity" /><field field_ref="woce" name="vovecrtz_SB002" long_name="sea water

→˓w velocity" /><field field_ref="utau" name="utau_SB002" long_name="surface

→˓downward x stress" /><field field_ref="vtau" name="vtau_SB002" long_name="surface

→˓downward y stress" /><field field_ref="ssh" name="ssh_SB002" long_name="sea

→˓surface height"/></field_group>


Note that these domain_ref locations are set in domain_ref.xml and are OK for AMM7

Resubmit:


Ah Ha! This runs but does not produce netCDF output. Good (because it can be fixed!)

Change filename:


→˓files -->"<file id="file51" name_suffix="_SB_grid_T" description="ocean T grid variables">

<field_group id="1h_SB001_grid_T" domain_ref="SB001" >

Nope

Switch fields (iodef.xml_fields_notworking):


→˓1h files -->"<file id="file0000" name_suffix="_SB_grid_T" description="ocean T grid

→˓variables"> (continues on next page)


<field_group id="1h_SB001_grid_T" domain_ref="SB001" ><field field_ref="sst" name="tos1" long_name="sea surface

→˓temperature" /></field_group><field_group id="1h_SB002_grid_T" domain_ref="SB002" ><field field_ref="sst" name="tos2" long_name="sea surface

→˓temperature" /></field_group>


Nope

Try something that did work (iodef.xml_field_working):

<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE." > <file id="file51" name_suffix="_grid_T" description="ocean T grid variables" ><field_group id="test1" domain_ref="SB001"><field field_ref="sst" name="tos1" long_name="sea surface temperature" /

→˓></field_group><field_group id="test2" domain_ref="SB002"><field field_ref="sst" name="tos2" long_name="sea surface temperature" /

→˓></field_group>


Now try and change stuff until it breaks (iodef.xml_file_id_):


→˓files -->"<file id="file51" name_suffix="_grid_T" description="ocean T grid variables" >

<field_group id="1h_SB001_grid_T" domain_ref="SB001" ><field field_ref="sst" name="tos1" long_name="sea surface

→˓temperature" /></field_group><field_group id="1h_SB002_grid_T" domain_ref="SB002" ><field field_ref="sst" name="tos2" long_name="sea surface

→˓temperature" /></field_group>


Spotted a spurious “ in the file_group definition line. Remove and try the tricky thing again:


→˓1h files --><file id="file0000" name_suffix="_SB_grid_T" description="ocean T grid


→˓potential temperature" />


7.2. Write two copies of same variable in different domains in single file 75



<file id="file001" name_suffix="_SB001_grid_T" description="ocean T grid→˓variables">

<field_group id="1h_SB001_grid_T" domain_ref="SB001" ><field field_ref="toce" name="thetao_SB001" long_name="sea water








→˓surface height"/></field_group><field_group id="1h_SB002_grid_T" domain_ref="SB002" ><field field_ref="toce" name="thetao_SB002" long_name="sea water
















→˓downward x stress" />(continues on next page)

7.3. AMM7 with two moorings files 77



<field field_ref="vtau" name="vtau_SB003" long_name="surface→˓downward y stress" />

<field field_ref="ssh" name="ssh_SB003" long_name="sea→˓surface height"/>


</file_group>

Edit domain_def.xml to include 3rd mooring (not sure where this is):: vi domain_def.xml <!– Test zoom forAMM7. Malin and North Sea –> <domain id=”SB001” zoom_ibegin=”0100” zoom_jbegin=”0250”zoom_ni=”2” zoom_nj=”2” /> <domain id=”SB002” zoom_ibegin=”0200” zoom_jbegin=”0250” zoom_ni=”2”zoom_nj=”2” /> <domain id=”SB003” zoom_ibegin=”0050” zoom_jbegin=”0050” zoom_ni=”2” zoom_nj=”2”/>

Resubmit:


THIS WORKS | walltime=00:06:14

Check this works with 1pt moorings. Edit domain_def.xml

Resubmit:


THIS WORKS, AS EXPECTED


7.4 Next steps:

Create mirrored AMM60 output with 100 moorings in one file and about 5 in another.

Create domain_def_1pt.xml with ipython notebook

Copy the 2 file iodef.xml file:

cd /work/n01/n01/jelt/gmaya/NEMO/CONFIG/XIOS_AMM7_nemo2/EXP00cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/→˓XIOS_AMM60_nemo/EXP_SBmoorings/iodef_sbmoorings_100moorings_2files.xml .cp iodef_sbmoorings_100moorings_2files.xml iodef.xml

Resubmit:


PENDING 27 Oct 2016. Expect one file with 100 moorings and one file with 5. These are 1pt moorings

ran for 59s



get NetCDF output but there is a problem with it

:: less NE198101.e4012898

terminate called after throwing an instance of ‘terminate called after throwing an instance of‘xios::CNetCdfExceptionxios::CNetCdfException’ ‘

what(): what(): Error in calling function nc_enddef(ncId)

NetCDF: HDF error Unable to end define mode of this file, given its id : 65536

Try a quick edit of iodef.xml. Change the file name to see if it then works. Use filenames file001 and file002, whichpreviously worked. <file id="file002" name_suffix="_SB034_grid_T" description="oceanT grid variables">

Resubmit:


PENDING 28 Oct 2016. Expect one file with 100 moorings and one file with 5. These are 1pt moorings

ran for 1min 1s. Same error with same file id: 65536

Comment out second file in iodef.xml

Resubmit:


Same error!

However it worked with 2 files and 3 moorings and a long time ago with one file and 30 (4pt) moorings.

Try one file with 33 moorings. 2nd file with 5 moorings.

Resubmit:

./run_nemo.sh annualrun.pbs 12 16 192 1981 1 1

Same but shorter error log:: terminate called after throwing an instance of ‘xios::CNetCdfException’ what(): Errorin calling function nc_enddef(ncId) NetCDF: HDF error Unable to end define mode of this file, given its id :65536

Remove the second file. Leaving one file with 33 (1pt) moorings. Resubmit:


Same error,

Resubmit with 20 (1pt) moorings:

7.4. Next steps: 79



This WORKS and runs beyond 2 mins.

Quit and resubmit with 25 (1pt) moorings:


WORKS6 min 41s to finish. Load balance looks OK

Resubmit with 30 (1pt) moorings:


Fails with new error after 2mins:

> Error [CNc4DataOutput::writeFieldData_ (CField* field)] : In file '/work/n01/n01/→˓jelt/xios-1.0_r703/src/output/nc4_data_output.cpp', line 1271 -> On writing field data: so_SB025In the context : nemoError in calling function ncPutVaraType(ncid, varId, start, count, op)NetCDF: HDF errorUnable to write value of a variable with id : 289

CONCLUSION. 25 moorings work but 30 moorings do not. Try 20 moorings in one file and 5 in a second.

Resubmit with 25 (1pt) + 5 moorings:


WORKED 6min 50s. CONCLUSION. 25 moorings work in one file but 30 moorings do not. However 25 moooringsin one file and 5 in another file also works.

Resubmit with 2 files each with 25 (1pt) moorings:


WORKED walltime=00:07:48 CONCLUSION. 25 moorings each in 2 files works. But 30 moorings in 1 file doesnot.

TEST Does 4pt mooring effect performance?

Resubmit with 2 files each with 25 (1pt) moorings:

cp domain_def_4pt.xml domain_def.xml./run_nemo.sh annualrun.pbs 12 16 192 1981 1 14014418.sdb

It worked walltime=00:09:00 CONCLUSION. 25 (1pt) moorings each in 2 files works. But 30 moorings in 1 filedoes not. Increasing from 1pt to 4pt moorings increases walltime from 7:45 to 9:00 mins



Test Use 1pt moorings for all the moorings spread across 133 files in groups of 25. Resubmit:

cp domain_def_1pt.xml domain_def.xmlcp iodef_sbmoorings_25moorings_133files.xml iodef.xml./run_nemo.sh annualrun.pbs 12 16 192 1981 1 14015212.sdb

Side line This looks like it is working, since it has not instantly crashed. Try 25moorings in 133 files with AMM60

EXCEEDED WALLTIME.

Trim simulation from 10 days to 1 day, keep the 20min queue:

vi annualrun.pbs#nit=$((10*tpd)) # 10 daysnit=$((1*tpd)) # 1 days

Resubmit:: ./run_nemo.sh annualrun.pbs 12 16 192 1981 1 1 4015274.sdb

Walltime exceeded. Cut down from 133 files to 13 files in iodef.xml Resubmit:


Walltime exceeded.


7.5 New Plan: Create a mask and output moorings as a 3D array

First check that AMM7 can compile:

https://www.evernote.com/shard/s523/nl/2147483647/086ee834-ae54-4384-8523-→˓79a1eee0d54e/

load modules:

module unload cray-hdf5 cray-netcdfmodule swap PrgEnv-cray PrgEnv-intelmodule load cray-hdf5-parallelmodule load cray-netcdf-hdf5parallel

Submit:

cd /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00./run_nemo.sh annualrun.pbs 12 16 192 1981 1 14017892.sdb

Plan is to emulate the changes Karen made to diagnose Internal Tides. These edits are all confined to diawri.F90First make a copy of working diawri.F90

/work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG/XIOS_AMM7_nemo/MY_SRC> cp diawri.F90 diawri.F90_30Oct16

Edit diawri.F90 (copy and paste in here when it works):

7.5. New Plan: Create a mask and output moorings as a 3D array 81


vi /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG/XIOS_AMM7_nemo/MY_→˓SRC/diawri.F90

...sbmask(161, 78) = 1sbmask(161, 79) = 1sbmask(160, 79) = 1sbmask(159, 79) = 1sbmask(158, 80) = 1

z3d(:,:,jpk) = 0.e0DO jk = 1, jpkm1

z3d(:,:,jk) = sbmask * un(:,:,jk)END DOCALL iom_put( "sb_u", z3d ) ! 3D u-velocityz3d(:,:,jpk) = 0.e0DO jk = 1, jpkm1

z3d(:,:,jk) = sbmask * vn(:,:,jk)END DOCALL iom_put( "sb_v", z3d ) ! 3D v-velocityz3d(:,:,jpk) = 0.e0DO jk = 1, jpkm1

z3d(:,:,jk) = sbmask * wn(:,:,jk)END DOCALL iom_put( "sb_w", z3d ) ! 3D w-velocity

z3d(:,:,jpk) = 0.e0DO jk = 1, jpkm1

z3d(:,:,jk) = sbmask * tsn(:,:,jk,jp_tem)END DOCALL iom_put( "sb_toce", z3d ) ! 3D temperaturez3d(:,:,jpk) = 0.e0DO jk = 1, jpkm1

z3d(:,:,jk) = sbmask * tsn(:,:,jk,jp_sal)END DOCALL iom_put( "sb_soce", z3d ) ! 3D salinityz3d(:,:,jpk) = 0.e0DO jk = 1, jpkm1

z3d(:,:,jk) = sbmask * fse3t(:,:,jk)END DOCALL iom_put( "sb_dept", z3d ) ! 3D T-point depth

CALL iom_put( "sb_utau", utau * sbmask ) ! 2D zontal wind stressCALL iom_put( "sb_vtau", vtau * sbmask ) ! 2D meridional wind stressCALL iom_put( "sb_ssh", sshn * sbmask ) ! 2D ssh

!ENDIF

Compile with -g -traceback:

vi /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/ARCH/arch-XC_ARCHER_INTEL.fcm...%FCFLAGS -integer-size 32 -real-size 64 -O3 -fp-model source -zero -fpp -→˓warn all -g -traceback%FFLAGS -integer-size 32 -real-size 64 -O3 -fp-model source -zero -fpp -→˓warn all -g -traceback



cd /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG./makenemo -n XIOS_AMM7_nemo -m ARCHER_INTEL

Copy executable:

cp XIOS_AMM7_nemo/BLD/bin/nemo.exe XIOS_AMM7_nemo/EXP00/.

Copy pbs script: # cp /work/n01/n01/jelt/gmaya/NEMO/CONFIG/XIOS_AMM7_nemo/EXP00/annualrun.pbs../XIOS_AMM7_nemo/EXP00/.

Edit paths in annualrun.pbs and run_nemo.sh for the appropriate execution directory:

vi annualrun.pbs # running for one dayvi run_nemo.sh

Edit field_def.xml:: UPDATE CHANGED field_ref to ID

vi field_def.xml

<field_group id=”sbmooring” > <field id=”sb_toce” name=”thetao”long_name=”sea_water_potential_temperature” unit=”degC” grid_ref=”grid_T_3D” /> <fieldid=”sb_soce” name=”so” long_name=”sea_water_salinity” unit=”psu” grid_ref=”grid_T_3D”/> <field id=”sb_u” name=”uo” long_name=”sea_water_x_velocity” unit=”m/s”grid_ref=”grid_U_3D” /> <field id=”sb_v” name=”vo” long_name=”sea_water_y_velocity”unit=”m/s” grid_ref=”grid_V_3D” /> <field id=”sb_w” name=”wo”long_name=”sea_water_z_velocity” unit=”m/s” grid_ref=”grid_W_3D” /> <fieldid=”sb_dept” name=”depth” long_name=”T-cell thickness” unit=”m” grid_ref=”grid_T_3D”/> <field id=”sb_ssh” name=”zos” long_name=”sea_surface_height_above_geoid”unit=”m” grid_ref=”grid_T_2D” /> <field id=”sb_utau” name=”tauuo”long_name=”surface_downward_x_stress” unit=”m/s^2” grid_ref=”grid_U_2D” /> <fieldid=”sb_vtau” name=”tauvo” long_name=”surface_downward_y_stress” unit=”m/s^2”grid_ref=”grid_V_2D” />

</field_group>

Edit iodef.xml:

vi iodef.xml<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE."> <file id="file51" name_suffix="SB" description="Shelf break moorings"><field_group group_ref="sbmoorings"/>

</file>...

</file_group>

Edit the wall time to since it appears the 24-48hr queue might be quicker (didn’t work. Not sure what queue is called):

vi annualrun.pbs#PBS -l walltime=00:20:00#PBS -q short

Submit:: cd /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00./run_nemo.sh annualrun.pbs 12 16 192 1981 1 1 4020211.sdb

PENDING (31 Oct 2016) cd /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00

• Does the output work?

7.5. New Plan: Create a mask and output moorings as a 3D array 83


• How fast / slow is it?

• How large is the output?

• Implement proper IF statements in diawri.F90

Assume all the above is correct. Copy to AMM60, compile and submit.

7.6 Problems with recompiled code. Start fresh with gmaya copy


vi iodef.xml

Edit to just have one file output

Edit annualrun.pbs to a 1 day simulation

Test code as is:


This works

Now try and make sbmooring diagnostic a 3D output in:


Copy edited source code:

cp /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG/XIOS_AMM7_nemo/MY_SRC/→˓diawri.F90 /work/n01/n01/jelt/gmaya/NEMO/CONFIG/XIOS_AMM7_nemo2/MY_SRC/diawri.F90

This has the sbmooring code (and Karen’s pycnocline depth) commented out.

load modules:

module unload cray-hdf5 cray-netcdfmodule swap PrgEnv-cray PrgEnv-intelmodule load cray-hdf5-parallelmodule load cray-netcdf-hdf5parallel

Compile this code:

cd /work/n01/n01/jelt/gmaya/NEMO/CONFIG./makenemo -n XIOS_AMM7_nemo2 -m ARCHER_INTEL

Copy executable:

cp XIOS_AMM7_nemo2/BLD/bin/nemo.exe XIOS_AMM7_nemo2/EXP00/.

Copy XML files:



cd /work/n01/n01/jelt/gmaya/NEMO/CONFIG/XIOS_AMM7_nemo2/EXP00#cp /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00/→˓iodef.xml iodef.xml#cp /work/n01/n01/jelt/src/NEMO_V3.6_STABLE_r6232/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00/→˓field_def.xml field_def.xml

Edit XML files:

vi iodef.xmlvi field_def.xml

Resubmit:


**PENDING 31 Oct 2016**

Next steps to uncomment sbmoorings code in .F90

7.6. Problems with recompiled code. Start fresh with gmaya copy 85

CHAPTER 8

AMM60 runs for ANChor

AIM:

• Make 2010 simulations of North Sea domain outputting hourly velocity, SSH and vertical viscosity for off-lineparticle tracking.

PATH: /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/

• This is a condensed version of the development process (AMM60 runs for ANChor - dev).

• There were two major problems: 1) generating mesh and mask files. 2) including AVM in the output.

• The former was solved with a clean build from a recent revision of v3.6_STABLE

• The latter was solved with a clean experiment from Karen’s code base.

26 Sept 2016

Create a clean experiment from Karen’s code base with AVM in the XML output. (copied old EXP_NSea toEXP_NSea_old):

cd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smagorsync -uartv EXP_NSea_prelauch_copy/ EXP_NSea

vi submit_nemo#PBS -N AMM60_NSea#PBS -l walltime=00:20:00..

$JOBDIR/finish_nemo.sh$JOBDIR/run_nemo

vi run_nemoexport RUNNAME=EXP_NSea..export nrestart_max=31


87

vi run_counter.txt1 1 7200 201001052 7201 14400 7200=201001093 14401 21600 14400=20100114

cp ../EXP_NSea2/iodef.xml .vi iodef.xml

<file_group id="1h" output_freq="1h" output_level="10" enabled=".TRUE."> 

<file id="file51" name_suffix="_NorthSea" description="T grid variables"><field_group id="NorthSea" domain_ref="NorthSea"><field field_ref="ssh" name="zos" long_name="sea_surface_

→˓height_above_geoid" /><field field_ref="avm" name="avm" long_name="vertical viscosity

→˓coefficient" /><field field_ref="uoce" name="uo" long_name="sea_water_x_

→˓velocity" /><field field_ref="voce" name="vo" long_name="sea_water_y_

→˓velocity" /><field field_ref="woce" name="wo" long_name="ocean vertical

→˓velocity" /></field_group>


(29 Sept 2016) Added in daily mean T and S:

<file_group id="1d" output_freq="1d" output_level="10" enabled=".TRUE."> <file id="file31" name_suffix="_grid_T" description="ocean T grid variables" ><field_group id="NorthSeaTS" domain_ref="NorthSea"><field field_ref="toce" name="thetao" long_name="sea_water_potential_

→˓temperature" /><field field_ref="soce" name="so" long_name="sea_water_salinity"

→˓ /></field_group></file>

</file_group>

vi domain_def.xml<domain_group id="grid_T">..<domain id="NorthSea" zoom_ibegin="0560" zoom_jbegin="0600" zoom_ni="560" zoom_nj=

→˓"600" />..

Submit:

./run_nemo

Check output is OK

Copy AMM60_1h_20100120_20100124_NorthSea.nc to SAN. Check it out with FERRET (use NX):

88 Chapter 8. AMM60 runs for ANChor


livljobs6% cd /scratch/jelt/tmp/livljobs6% scp [email protected]:/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/→˓AMM60smago/EXP_NSea/OUTPUT/AMM60_1h_20100120_20100124_NorthSea.nc .

IS OUTPUT OK? All looks fine (also u,v,w not shown)

21 Oct 2016

Extend the run through Aug 2010:

vi run_counter.txt...30 208801 216000 208800=2010052931 216001 223200 216000=2010060332 223201 230400 223200=2010060833 230401 237600 230400=2010061334 237601 244800 237600=2010061835 244801 252000 244800=2010062336 252001 259200 252000=2010062837 259201 266400 259200=2010070338 266401 273600 266400=2010070839 273601 280800 273600=2010071340 280801 288000 280800=2010071841 288001 295200 288000=2010072342 295201 302400 295200=20100728

Add 6 more integrations on. Need only change nrestart_max:

vi run_nemoexport nrestart_max=47

Submit:


NOTE: I HAVE NOT EDITTED THIS run_nemo FILES WHICH MAY INCORRECTLY sed SOME VARIABLESCHECK /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmIT/EXP_harmIT/run_nemo OR /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings/run_nemo OR /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings2/run_nemo

Time step does not necessarily correspond to the end timestep in run_counter.txt:: cd/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/WDIR more time.step

309420

tail ../run_counter.txt 34 237601 244800 237600=20100618 35 244801 252000 244800=20100623 36 252001259200 252000=20100628 37 259201 266400 259200=20100703 38 266401 273600 266400=2010070839 273601 280800 273600=20100713 40 280801 288000 280800=20100718 41 288001 295200288000=20100723 42 295201 302400 295200=20100728 43 302401 309600 302400=20100802

89


Though this run was terminated because the wall time was exceeded. It is perhaps possilbe that the time.step ran awayas things were getting shut down?

Created:

AMM60_1d_20100729_20100802_grid_T.ncAMM60_1h_20100729_20100802_NorthSea.nc

BUT Wall time exceeded. Resubmit:: run_nemo 4004867.sdb

How is output? Check ‘‘/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUT‘‘

Two new runs:

-rw------- 1 jelt n01 688183877 Oct 21 21:41 AMM60_1d_20100803_20100807_grid_T.nc-rw------- 1 jelt n01 33065238457 Oct 21 21:41 AMM60_1h_20100803_20100807_NorthSea.nc-rw------- 1 jelt n01 74882 Oct 21 21:42 AMM60_NSea.o4004867-rw------- 1 jelt n01 688183877 Oct 21 22:03 AMM60_1d_20100808_20100812_grid_T.nc-rw------- 1 jelt n01 33065238457 Oct 21 22:03 AMM60_1h_20100808_20100812_NorthSea.nc

WDIR/stdouterr: Job killed The wall time was exceeded.

Plan: re calculate AMM60_1h_20100813_20100817_NorthSea.nc

vi run_counter.txt:: . . . 44 309601 316800 309600=20100807 45 316801 324000 316800=20100812

Increase wall time:


Resubmit:

run_nemo4005690.sdb

PENDNG 22 Oct 2016. How is output? Check ‘‘/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUT‘‘

8.1 New subdomain for North Sea, Jan 2010 - Nov 2013

12 Dec 2016

Exisiting subdomain does not capture all the Lophelia platforms. Aim: Move northern boundary to 64N

PATH: /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/

vi run_counter.txt:

1 1 7200 201001052 7201 14400 7200=20100109

Link in restart:

ln -s /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/→˓CONFIG/AMM60smago/EXP_notradiff/RESTART/00007200 /work/n01/n01/jelt/NEMO/NEMOGCM/→˓CONFIG/AMM60smago/EXP_NSea/RESTART/.


Only need to run to end of Apr 2010 vi run_nemo:

export nrestart_max=25

vi submit_nemo.pbs:

#PBS -A n01-NOCL

Move the northern boundary north (use AMM60_extract_bathy_subdomain.ipynb) vi ../EXP_NSea/domain_def.xml:

<domain id="NorthSea" zoom_ibegin="0560" zoom_jbegin="0600" zoom_ni="560" zoom_nj="766→˓" />

Edit run_counter.txt to accomodate this restart:: vi run_counter.txt

Submit:


**PENDNG 12 Dec 2016. How is output? Check ``/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/→˓AMM60smago/EXP_NSea/OUTPUT``. Should run to Apr**

Resubmit 20 Dec 2016:


**PENDNG 20 Dec 2016. How is output? Check ``/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/→˓AMM60smago/EXP_NSea/OUTPUT``. Should for ~50 steps**

50 restarts gets to September: 20100911

Edit run_nemo for total of 100 restarts and resubmit:


PENDNG 22 Dec 2016. How is output? Check ‘‘/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUT‘‘.Should for ~50 steps

24 Dec 2016 Quick turnaround on the simulation. Done up to 19/5/11 Add another 100 timesteps. Resubmit:



26 Dec 2016 Quick turnaround. Done up to 20120218. Edit run_nemo to maxcount 300 and resubmit:



Exceeded walltime at end of Nov 2013. Resubmit:

8.1. New subdomain for North Sea, Jan 2010 - Nov 2013 91



Hit walltime. Increase walltime to 30mins from 25mins:



PENDNG 13 Jan 2017. How is output? Check: ‘‘cd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUT‘‘

—

(15 Feb 2017) Output file was lost or corrupted:

ncdump -h /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUT/AMM60_1h_→˓20100204_20100208_NorthSea.ncncdump: /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUT/AMM60_1h_→˓20100204_20100208_NorthSea.nc: NetCDF: Unknown file format

Recreate it. Edit run_counter.txt, run_nemo and check submit_nemo.pbs:

cp run_counter.txt run_counter.txt_finished

vi run_counter.txt1 1 7200 201001052 7201 14400 7200=201001093 14401 21600 14400=201001144 21601 288005 28801 36000 28800=201001246 36001 43200 36000=201001297 43201 50400 43200=20100203

vi run_nemoexport nrestart_max=300-->export nrestart_max=7

Submit:


PENDING 15 Feb 2017

8.2 Compression of output

Compress output data using the serial queue. Instead of trying to use netcdf compression, use gzip:

cd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/

less gzip2012#!/bin/bash#PBS -N AMM60gz12





#PBS -l select=serial=true:ncpus=1#PBS -l walltime=24:00:00#PBS -o AMM60gz12.log#PBS -e AMM60gz12.err#PBS -A n01-NOCL####################################################set up paths

cd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUTgzip AMM60_1?_2012*nc

# qsub -q serial <filename>############################

Then submit the jobs:

qsub -q serial gzip2012qsub -q serial gzip2013

----

(22 May 2017)

8.3 Unzip jun-aug for 2010-2013 for Locate project

cd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUT

qsub -q serial gunzip2010_may_aug4527147.sdbjelt@eslogin004:/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea> qsub -q→˓serial gunzip2011_may_aug4527148.sdbjelt@eslogin004:/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea> qsub -q→˓serial gunzip2012_may_aug4527149.sdbjelt@eslogin004:/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea> qsub -q→˓serial gunzip2013_may_aug4527150.sdb

—

8.4 Sharing with Matthew Toberman

Prepare files for JASMIN. Hourly U,V files Daily T,S files for the North Sea.

Zip any unzipped files:

jelt@eslogin006:/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea> qsub -q→˓serial gzip20104796211.sdbjelt@eslogin006:/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea> qsub -q→˓serial gzip2011


8.3. Unzip jun-aug for 2010-2013 for Locate project 93



4796212.sdbjelt@eslogin006:/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea> qsub -q→˓serial gzip20124796214.sdbjelt@eslogin006:/work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea> qsub -q→˓serial gzip2013

Copy files to JAMIN nemo workspace:

ssharcherexec ssh-agent $SHELL(this produces an error but it works)ssh-add ~/.ssh/id_rsa_jasmincd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea/OUTPUTrsync AMM60_2013_grid_T.tar [email protected]:/group_workspaces/jasmin2/→˓nemo/vol5/public/AMM60/pycnmix/.

Notes on Evernote. Need to copy here when ready.


CHAPTER 9

AMM7 3D Harmonic build and submission

9.1 0. Plan

• Copy directory from Maria. Run

• Compile and run using Maria’s methods

• Reference Maria’s codebase to a fixed revision of v3.6_STABLE

• Compile and run

• Remove initial straight copied version

• Update to AMM60 configuration.

9.2 1. Copy and run code from Maria

Create empty directory tree:

~/work/from_mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00

Synchronise EXP00 dir:

rsync -uartv --delete /work/n01/n01/mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/→˓EXP00/ ~/work/from_mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00

NB /work/n01/n01/mane1/AMM7_w is the 3D harmonic experiment. The alias points to /→˓work/n01/n01/mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00

Edit submission script paths:

vi subm#RUNDIR=/work/n01/n01/mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00RUNDIR=/work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00


95

vi rsub#RUNDIR=/work/n01/n01/mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00RUNDIR=/work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00

Fix a bug in the submission script. Comment out ndate redefinition on line 119:

cd /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00vi subm...#ndate=$yy$month1$day

Manually fix the restart files to appear. (Perhaps this should not be done in this script as it will get repeated witheach iteration):

cd /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00for i in {0..9}; do cp 'files_restart_198112/restart_000'$i.nc 'restart_000'$i.nc;→˓ donefor i in {10..99}; do cp 'files_restart_198112/restart_00'$i.nc 'restart_00'$i.nc;→˓donefor i in {100..191}; do cp 'files_restart_198112/restart_0'$i.nc 'restart_0'$i.nc;→˓done

Change bathy file:

cd /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00ln -s bathy_meter_orig.nc bathy_meter.nc

Use Maria’s executable:

vi subm...export EXEC=opa_harm3d

Comment out `difvho, avt, variable from output (it was associated with an error when things weren’t working andI haven’t reinstated it):

vi iodef.xml<file id="file25" name_suffix="_grid_W" description="ocean W grid variables" >

<field field_ref="e3w" /><field field_ref="gdepw" /><field field_ref="woce" name="wo" long_name="ocean vertical velocity" /

→˓>

</file>

Submit:

./rsub subm 1982 1 1qsub -v m=1,y=1982,nit0=1,ndate=19820101 -o /work/n01/n01/jelt/from_mane1/V3.6_ST/→˓NEMOGCM/CONFIG/XIOS_AMM7_nemo/EXP00/GA-AMM7-1982-01 -N GA198201 subm4196147.sdb

This runs. AND the output looks sensible for TKE25H, EPS25H and M2X_SSH.

96 Chapter 9. AMM7 3D Harmonic build and submission


9.3 2. Compile new executable

Copy source code from Maria:

cd /work/n01/n01/mane1/V3.6_ST/NEMOGCMrsync -uart ARCH/ /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/ARCHrsync -uart EXTERNAL/ /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/EXTERNALrsync -uart fcm-make/ /work/n01/n01/jelt/from_mane1/V3.6_S/fcm-makersync -uart NEMO/ /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/NEMOrsync -uart SETTE/ /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/SETTErsync -uart TOOLS/ /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/TOOLScp License_CeCILL.txt /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/.cp CONFIG/XIOS_AMM7_nemo/cpp_XIOS_AMM7_nemo.fcm /work/n01/n01/jelt/from_mane1/V3.6_ST/→˓NEMOGCM/CONFIG/XIOS_AMM7_nemo/.

Add compiler tide harmonic key (key_diaharm), remove key_gen_IC:

cd /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCMvi CONFIG/XIOS_AMM7_nemo/cpp_XIOS_AMM7_nemo.fcm

bld::tool::fppkeys key_dynspg_ts key_ldfslp key_zdfgls key_mpp_mpi \key_netcdf4 \key_nosignedzero key_traldf_c2d \key_dynldf_c2d key_bdy key_tide key_vvl key_iomput \key_diaharm

XIOS executables differ between that which I and Maria had previously used. Either using James’ or Andrew build:

cd /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/ARCHdiff arch-XC_ARCHER_INTEL.fcm /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_→˓BDY_VERT_INTERP/NEMOGCM/ARCH/arch-XC_ARCHER_INTEL.fcm35c35< %XIOS_HOME /work/n01/n01/acc/XIOS_r484---> %XIOS_HOME /work/n01/n01/jdha/ST/xios-1.0

James’ is newer. Use his:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/ARCH/→˓arch-XC_ARCHER_INTEL.fcm /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/ARCH/.

Compile:


cd /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/CONFIG



Copy executable to EXPeriment directory:

cp XIOS_AMM7_nemo/BLD/bin/nemo.exe XIOS_AMM7_nemo/EXP00/opa_harm3d

Edit subm to use new executable:

9.3. 2. Compile new executable 97


vi subm...export EXEC=opa_harm3d

Copy fresh restart files:


Submit:


This WORKS!! AND the output looks sensible for TKE25H, EPS25H and M2X_SSH.

PLAN

Now to replicate Maria’s code need to create a restart in the summer. Say end of May. Then do a 3 month tidalanalysis, for June, July, Aug. So need a 5 month simulation. Launch 5 successive restarts from Jan 2012. Save the lastrestart set.

Copy fresh restart files from Dec 2011:


Edit the number of timestep in the simulation to do one month. Comment out the test cases:

vi subm...#nit=576 # 2 days#nit=4320 # 15 days#nit=25920 # 90 days

For 2012 the namelist comes from namelist_cfg.template_skag_climate. The harmonic analysis is keptat 15 days because we don’t care during spin up to June. THIS WILL NEED TO BE CHANGED:

vi namelist_cfg.template_skag_climate...nitend_han = 4320

Edit the iodef.xml. Comment out the daily U,V,T,W,tides files. Otherwise a month will not complete on the shortqueue (Just M2 ran for 30days in 20mins). Lots of consituents commented out. Again this will need to be changed forthe production run.



Submit run:


Jan completed. Restart for Feb (on standard queue after 8pm):


Feb completed in 15mins on standard queue. Switch back to 20 mins on short queue:




Backup the restarts from end of May 2012:

mkdir files_restart_201205cp restart_????.nc files_restart_201205/.

Configure simulation to run for 3 months with all the XIOS output:

vi iodef.xml

*restore outputs*

vi namelist_cfg.template_skag_climatenitend_han = 25920 ! 105120 ! 210528 ! Last time step used for harmonic analysis

vi subm#PBS -l walltime=03:00:00##PBS -q short...nit=25920 # 90 days

Submit 3 month run:


The 3 month run took walltime-01:22:51. A quick look with Ferret. Tides appear OK.

9.3. 2. Compile new executable 99


ACTION: check grid_W.nc output

THIS IS THE ACTUAL END OF WHERE I HAD A THREE MONTH 3D TIDAL RUN WITH AMM7

(17 May 2017) Submit a 2 month run for comparison with AMM60:

vi submnit=17568 # 61 days

vi namelist_cfg.template_skag_climatenn_msh = 0...nitend_han = 17568 ! 25920 ! 105120 ! 210528 ! Last time step used for harmonic→˓analysis


This breaks because the boundary forcing data has moved

9.4 3. Other things learnt while debugging

Could submit again with non-unit nit0, so that calendar parameters are read from restart e.g.:

./rsub subm 2003 8 2

Edit the namelist parameters to do short queue runs. e.g. Use climatology namelist EXCEPT for if [ $yy -ge1990 -a $yy -le 2009 ]; then:

Run time 15 days (Cut down iodef.xml output so 1 month should now work):

vi submnit=4320 # 15 days

Harmonic analysis duration 15 days (Cut down iodef.xml output so 1 month should now work):

vi namelist_cfg.template_skag_climatenitend_han = 4320

9.5 4. Identify code changes in Maria’s source

Create a patch file for changes in the OPA_SRC directory. In the following there are duplicated lines for two differentrevisions. I wasn’t sure if a later revision would be preferable as it might have fewer changes. I think it just hasdifferent changes. . . In implementing the following either follow r6204 OR r7564, not both.

Setup some directory aliases:

export TDIR=/work/n01/n01/jelt/TRYexport SDIR=/work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRCexport WDIR=/work/n01/n01/jelt



Clean out the TRY directory:

cd $TDIRrm -rf *

Checkout a v3.6_STABLE at revision 6204/7564. (This corresponded to the highest version number in Maria’s code):

svn co http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE@6204svn co http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE@7564

Create a link to the clean files in modified source:

ln -s $TDIR/nemo_v3_6_STABLE/NEMOGCM/NEMO/OPA_SRC/ $SDIR/../OPA_SRC_r6204ln -s $TDIR/nemo_v3_6_STABLE/NEMOGCM/NEMO/OPA_SRC/ $SDIR/../OPA_SRC_r7564

For some reason OPA_SRC is recursively defined inside the target dir (for r6204):

rm $SDIR/../OPA_SRC_r6204/OPA_SRC

Using the syntax: diff -crB OrigSrc ModSrc > ModSrc.patch create a patch file, where -c context, -rrecursive (multiple levels dir), -B is to ignore Blank Lines, -N new files. (Following http://linux.byexamples.com/archives/163/how-to-create-patch-file-using-patch-and-diff/)

Create a patch only for *90 files. This is neatly done by creating a list of files that are not *90 files and excluding themfrom the diff (diff has no include option) I didn’t do this for the r6204 patch:

cd $SDIR/..find OPA_SRC -type f | grep --text -vP "90$" | sed 's/.*\///' | sort -u > file1find OPA_SRC_r7564/ -type f | grep --text -vP "90$" | sed 's/.*\///' | sort -u >>→˓file1sort -u file1 > excludelistrm file[12]

diff -crBN OPA_SRC_r6204 OPA_SRC > OPA_SRC_r6204_harm3d.patchdiff -crBN OPA_SRC_r7564 OPA_SRC -X excludelist > OPA_SRC_r7564_harm3d.patch

Make a new ‘test’ directory to patch:

rsync -ravt /work/n01/n01/jelt/TRY/nemo_v3_6_STABLE/NEMOGCM/NEMO/OPA_SRC/ OPA_SRC_→˓harm3dcd OPA_SRC_harm3d/

Do a dry run first:

patch --dry-run -p1 -i ../OPA_SRC_r6204_harm3d.patchpatch --dry-run -p1 -i ../OPA_SRC_r7564_harm3d.patch

This is a bit heavy handed and patches all the files, including svn logs and things that do not need patching. Howevera diff -r OPA_SRC_harm3d/ $SDIR returns a set of empty lines.

9.6 5. Check for differences between AMM7 and AMM60

9.7 6. Build harmonic code from clean checkout

Load modules:

9.6. 5. Check for differences between AMM7 and AMM60 101

http://linux.byexamples.com/archives/163/how-to-create-patch-file-using-patch-and-diff/

http://linux.byexamples.com/archives/163/how-to-create-patch-file-using-patch-and-diff/


module swap PrgEnv-cray PrgEnv-intelmodule load cray-netcdf-hdf5parallelmodule load cray-hdf5-parallel

Setup some directory aliases:

export WDIR=/work/n01/n01/jeltexport CDIR=$WDIR/NEMO/nemo_v3_6_STABLE_r7564_harm3d/NEMOGCM/CONFIG

Clean out the TRY directory:

cd $WDIR

Checkout a v3.6_STABLE at revision 7564. (This is currently the latest v3.6_STABLE revision available):

svn co http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE@7564

Change checked out repo directory name. Have to use suffix harm3d since the source code in OPA_SRC will bemodified from checked out version:

mv nemo_v3_6_STABLE/ NEMO/nemo_v3_6_STABLE_r7564_harm3d

Copy ARCH file:

cp /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/ARCH/→˓arch-XC_ARCHER_INTEL.fcm $CDIR/../ARCH/.

Compile only with OPA_SRC:

cd $CDIR./makenemo -n XIOS_AMM60_nemo -m XC_ARCHER_INTEL -j 10

Fails. Remove spurious key_lim2, replace with the following (for AMM7):

vi XIOS_AMM60_nemo/cpp_XIOS_AMM60_nemo.fcmbld::tool::fppkeys key_dynspg_ts key_ldfslp key_zdfgls key_mpp_mpi \

key_netcdf4 \key_nosignedzero key_traldf_c2d \key_dynldf_c2d key_bdy key_tide key_vvl key_iomput \key_diaharm

Does not work

Now have a MY_SRC directory. Copy original source code and apply patch there (exclude /.svn folders):

rsync -arvt --exclude=".*" $WDIR/NEMO/nemo_v3_6_STABLE_r6204/NEMOGCM/NEMO/OPA_SRC/*→˓XIOS_AMM60_nemo/MY_SRC/.

Obtain and apply patch:

cd $CDIR/XIOS_AMM60_nemocp /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC_r7564_harm3d.patch .

cd MY_SRCpatch --dry-run -p1 -i ../OPA_SRC_r7564_harm3d.patch



The BDY patches seemed OK but all the others had issues. Could apply patch directly at OPA_SRC directory.

END: DOES NOT WORK.

Apply patch directly at OPA_SRC:

cp /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/NEMO/OPA_SRC_r7564_harm3d.patch→˓$CDIR/../NEMO/.cd $CDIR/../NEMO/OPA_SRCpatch --dry-run -p1 -i ../OPA_SRC_r7564_harm3d.patch

If it doesn’t return errors:

patch -p1 -i ../OPA_SRC_r7564_harm3d.patch

This works. But does not preserve the source code.

(19 Jan 2017)

Copy in Fred’s bdy code into MY_SRC. (Came in an email from Maria). Files to be found at:

/work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r7564_harm3d/NEMOGCM/CONFIG/XIOS_AMM60_nemo/→˓MY_SRC/bdydyn3d.F90/work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r7564_harm3d/NEMOGCM/CONFIG/XIOS_AMM60_nemo/→˓MY_SRC/bdyini.F90

Build again:

cd $CDIR./makenemo clean./makenemo -n XIOS_AMM60_nemo -m XC_ARCHER_INTEL -j 10

This produces a new executables, which was the same as the one generated from Maria’s codebase Except for changeswith bdydyn3d.F90 and bdyini.F90 and modifications she made to nemogcm.F90 before I copied it.

Attempting to compile throws up an error associated with the sponge_factor array. Fixing it would mean fiddlingwith some more bdy files. If I don’t need the sponge then I can just comment out this line. To get the code to build Icommented out the line starting sponge_factor. This is OK-ish if the ln_sponge is not true, which it was not inMaria’s run. (But was in Karen’s)..

PLAN:

• Copy new executable into /work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_AMM7_nemo/ and try it out (ideally once the summer restarts are generated)

9.7. 6. Build harmonic code from clean checkout 103

CHAPTER 10

Diagnostics of 3D Harmonics

(3 Feb 17)

ACTIONS:

• move good data from /scratch/tmp to /projectsa

• figure out what supporting grid data is also needed.

• AMM60 3 month run?

• How many tidal harmonics do I need to use (NB I don’t need to output them all)

Latex notes: ~/Documents/my/IT_AMM60_16/IT_AMM60.pdf

iPython notebook: ~/python/ipynb/NEMO/internaltideharmonics_NEMO.ipynb

For analysis of data on ARCHER, must run notebook on livljobs because sshfs only works on this machine.

Once data is checked and OK bring it to the SAN:

#e.g. for 3 months of AMM7 data (June-Aug 2012)rsync -uartv [email protected]:/work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/→˓CONFIG/XIOS_AMM7_nemo/EXP00/GA_1d_20120601_20120829*nc /scratch/jelt/tmp/.scp [email protected]:/work/n01/n01/jelt/from_mane1/V3.6_ST/NEMOGCM/CONFIG/XIOS_→˓AMM7_nemo/EXP00/coordinates.nc /scratch/jelt/tmp/coordinates_AMM7.nc

# e.g. 5 days of AMM60 datarsync -uartv [email protected]:/work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_→˓NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmIT2/EXP_harmIT2/OUTPUT/→˓AMM60*nc /scratch/jelt/tmp/.scp [email protected]:/work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_→˓VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmIT2/EXP_harmIT2/WDIR/coordinates.nc /→˓scratch/jelt/tmp/coordinates_AMM60.nc

which can be analysed on livljobs6. Currently holding data on /scratch/jelt/tmp. This is not a good place. It shouldmove to /projectsa at some point.

Some data exists on a USB drive for protyping on the move.

105


106 Chapter 10. Diagnostics of 3D Harmonics

CHAPTER 11

Introduction

This file is the launch page for different work flows.

Need to add more stuff

External page

107

http://nemo-docs.readthedocs.io/en/latest/?


108 Chapter 11. Introduction

CHAPTER 12

3D Harmonic analysis

Logs at 3D Harmonic analysis

PATH: /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo_harmIT

PROGRESS:

• Successfully compiled Karen’s code.

• Ran for 20 mins in a clean new EXP config. Ran as EXP_NSea.

• Checked output is OK for NSea test: AMM60_read_plot.ipynb

• Successfully compiled and ran Karen’s code with Maria’s 3D harmonics (though w/o key_diaharm). Outputtedvirtual moorings OK.

• Recompiled Maria’s 3D harmonics with key_diaharm

• Added in new field_def.xml variables for the 3D harmonics

• Ported dev logging to GitHub: https://github.com/jpolton/EXP_harmIT.git

• Edits to namelist_cfg: nnit???_han

• Exceed walltime: 2day in 5mins (1.5days in 5mins)

• 5 day (19mins) simulation with 3D harmonic output WORKS (22 Oct 2016)

• Found bug in namelist_cfg edit with nn_write

• Added ln_dia25h and edited harmonic constituents wanted in namelist_cfg.

• Added 25hour diagnostics to field_def.xml

• Trying 25hour output diagnostics with 3D harmonics. It appears that the dia25h does activate.

CURRENT:

• Try mapping AMM60 to v3.6_STABLE from AMM7 . PENDING 16 Jan 2017

109

3D_Harmonic_analysis.html

https://github.com/jpolton/EXP_harmIT.git

AMM7_3D_Harmonic_analysis.index


• Start analysising output with ‘diagnostics_3D_harmonics<diagnostics_3D_harmonics.html>‘_. PENDING3 Feb 2017

PLANS:

• I need to spin up this model to get meaningful output. Perhaps switch it in for the ANChor runs? Talk to Maria

• Changes to field_def.xml - with 3D harmonics no longer using Karen’s version. Need to propagate throughnotes.

• Spring neaps variation. Seasonal variation. Check notes elsewhere for plans

• Write some papers.

110 Chapter 12. 3D Harmonic analysis

CHAPTER 13

AMM60 June 2012 restarts and virtual mooring output

Logs at AMM60 restarts and XIOS dev. Restarts on the new compiled code.

PATH: /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo

PROGRESS:

1. Successfully compiled Karen’s code.

2. Generate the XML code using python notebook: jcomp_tools_dev/AMM60_build_iodef_from_latlon.ipynb

3. Successfully generated virtual mooring test data for early June 2012 using Karen’s restart: /work/n01/n01/kariho40/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/AMM60smago/EXPD376/

4. Generated virtual moorings for Jo. Got the green light to proceed with high resolution data.

5. Found a bug in the iodef.xml file using AMM7

6. 3305 XML files create memory problem

7. 3305 moorings in 1 file completed but no XML output. (with or without XML typo)

8. 2 moorings in 1 files works and outputs (5days takes 20mins)

9. 999 moorings in 1 file works and outputs (27hours takes 20mins) ACTION. Check output. I LOST THEDATA

10. 100 moorings each in 34 files (OOM after 25.1hrs in 10mins. No OUTPUT) BUG IN iodef.xml

11. 100 moorings each in 34 files (doubled XIOS processors) Netcdf issues

12. Found error in namelist_cfg with nn_write not being edited

13. 100 moorings each in 34 files (fewer XIOS processors per node) PENDING (21 Oct 2016)

14. 100+5 moorings each in 1+1 files (fewer XIOS processors per node) PENDING (21 Oct 2016)

111

AMM60_restarts_and_XIOS_dev.html


15. 100+5 moorings each in 1+1 files (fewer XIOS processors per node). EXP_SBmoorings3. In old executable.At least 40mins for 1 day Wall time exceeded

16. 100+5 moorings each in 1+1 files (x10 XIOS processors per node). EXP_SBmoorings3. In old executable.EXCEEDED RESOURCE ALLOWED

17. 100+5 moorings each in 1+1 files. 1pt moorings. (fewer XIOS processors per node). EXP_SBmoorings. 20minqueue BROKE

18. 3 moorings in 2 files works in AMM7.

19. Switching to outputting 3d array for moorings output usugn a 2d mask array for moorings locations

20. Shelf break mooring subdomain. Output 3D arrays with a mooring mask applied.

21. June 2012 simulation finished

22. off-line compression on netcdf output. Gzipping saves approximately a factor of 2 space.

CURRENT: cd /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_SBmoorings

ACTIONS: (16 Jan 2016)

• Tidy up and close this work Object.

ISSUES:

• Not succeeded in outputting 3305 1d moorings. An I/O problem?

• Outputted using masked 3D spatial arrays.

PLANS:

• Migrate tech notes to NEMO skills

• Gzip output in the finish script?

• Other output: * Slowly varying stratification * Baines forcing * . . .

112 Chapter 13. AMM60 June 2012 restarts and virtual mooring output

CHAPTER 14

NEMO skills

Aim: document basic functionality in getting stuff setup and working.

• Compiling AMM60.

• Generate mesh and mask files on short queue (originally for AMM60 ANChor output).EXP_NSea_mesh. Doesn’t actually work. Tried reducing tilmestep to 6s. No good. (22 Sept16)

• Configure AMM60 on the short queue - though this doesn’t actually work. . .

• Generate mesh and mask files on long queue (Karen’s codebase. Restart. Fresh compile.nn_msh=[1,3]). Doesn’t work

• Generate AMM60 mesh and mask files from stable build (EXP_meshgen) qsubnc_mesh_build.pbs --> mesh_mask.nc. WORKS!

• Compile -g -traceback debug version - there are some notes somewhere, though there is notmuch to say.

• Debugging XML sensitivities in the short queue with AMM7

PATH: /work/n01/n01/jelt/NEMO/NEMOGCM_jdha/dev_r4621_NOC4_BDY_VERT_INTERP/NEMOGCM/CONFIG/XIOS_AMM60_nemo/ - DIDN’T WORK /work/n01/n01/jelt/NEMO/nemo_v3_6_STABLE_r6939/NEMOGCM/CONFIG/XIOS_AMM60_nemo/EXP_meshgen - DIDWORK (22 Sept 2016)

PROGRESS:

• Compiled code (Karen’s code base)

• Built AMM60 on short queue (as a cold start) - does not run

• Compile code in debug -g -traceback mode

• Built v3.6_STABLE. Restart with nn_msh = 1 --> mesh_mask.nc

CURRENT:

Issues:

113


https://www.evernote.com/shard/s523/nl/2147483647/f0149755-b99c-4f5c-a488-bd2940e16d20/

https://www.evernote.com/shard/s523/nl/2147483647/e0445ece-518f-4a5f-aff5-6ff5df8a491e/

https://www.evernote.com/shard/s523/nl/2147483647/300e814a-5c06-4677-9c2e-d76e02133a11/

Generate_AMM60_mesh_and_mask_files_from_stable_build.html

Debugging_XIOS_in_AMM7.html


• Generating meshes is a problem with Karen’s code. It worked with a recent v3.6_STABLE.

• Can’t run short queue AMM60

Conclusions

• Can generate mesh_mask.nc file with new stable code but not Karen’s code.

• Cannot run on the short queue (SEG FAULTS). It might work on v3.6_STABLE

PLANS:

• Try short queue with v3.6_STABLE

• Write compiling debug executable as a note

• Tidy up short queue pages.

• Compress output in the finish script?

114 Chapter 14. NEMO skills

CHAPTER 15

ANChor runs

dev logs at AMM60 runs for ANChor - devProduction logs at AMM60 runs for ANChor

PATH: /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/

PROGRESS:

• Running with hourly velocity output with vertical viscosity AVM

• mesh files generated with Generate AMM60 mesh and mask files from stable build

• Have output through to the end of Aug 2012 (27th)

• Generated hourly output for North Sea: Jan 2010 - Nov 2013

CURRENT:

• Finished integration production runs AMM60 runs for ANChor cd /work/n01/n01/jelt/NEMO/NEMOGCM/CONFIG/AMM60smago/EXP_NSea (22 Oct 2016).

ACTIONS:

• gzipping 2012 and 2013 Submitted (16 Jan 2017)

PLANS:

• Gzip compression of output in the finish script

115

https://www.evernote.com/shard/s523/nl/2147483647/0e6c9337-5fa1-4039-9af8-00e7e02f2daf/

AMM60_runs_for_ANChor.html


AMM60_runs_for_ANChor.html


116 Chapter 15. ANChor runs

CHAPTER 16

AMM60 in v3.6_STABLE

Updating Karen’s code base. Production logs at AMM60 in v3.6_STABLE

PROGRESS:

• downloaded and compiled v3.6_STABLE. Fresh build with v3.6_STABLE

• generated AMM60 mesh files. Generate AMM60 mesh and mask files from stable build

• Tried building AMM60 in AMM7 environment: AMM60 from AMM7v3.6_STABLE environment (18 Jan 17).Not succesful

• Tried building AMM60 with new source code (19-24 Jan 17). Not successful

CURRENT:

• Have not updated AMM60 to v3.6_STABLE. Compiling AMM60 from v3.6_STABLE. In progress

• Copied Maria’s 3D harmonics version of AMM7 running on v3.6_STABLE: AMM7 3D Harmonic analysis INPROGRESS (11 Jan 17)

117

AMM60_v3_6_STABLE.html

https://www.evernote.com/shard/s523/nl/2147483647/408b131e-fddf-4fee-94ec-d4f0671f0e21/


AMM60_from_AMM7v3_6_STABLE.html

AMM60_v3_6_update.html

https://www.evernote.com/shard/s523/nl/2147483647/cc65c445-f129-4db7-9310-b976c49645b0/

AMM7_3D_Harmonic_analysis.html


118 Chapter 16. AMM60 in v3.6_STABLE

CHAPTER 17

Indices and tables

• genindex

• modindex

• search

119