Download - Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS
![Page 1: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/1.jpg)
Training session 2 :Advanced training course on modipsl and libIGCM
November 14th 2013, MdS
![Page 2: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/2.jpg)
Outline
• IPSL climate modelling centre (ICMC) presentation
• IPSLCM history and perspective • Mini how to use modipsl/libIGCM• Post-processing with libIGCM• Monitoring a simulation• Hands-on
![Page 3: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/3.jpg)
IPSL climate modelling centre (ICMC)
![Page 4: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/4.jpg)
Modeling platform (IPSL-ESM)Arnaud Caubel (LSCE) - Marie-Alice Foujols (IPSL)
Data Archive and Access RequirementsSébastien Denvil (IPSL) - Karim Ramage (IPSL)
Atmospheric and surface physics and dynamics (LMDZ)
Frédéric Hourdin (LMD) - Laurent Fairhead (LMD)
Ocean and sea ice physics and dynamics (NEMO, LIM)
C Ethé (IPSL) - Claire Lévy - Gurvan Madec (LOCEAN)
Atmosphere and ocean interactions (IPSL-CM, different resolutions)
Sébastien Masson (LOCEAN) - Olivier Marti (LSCE)
Biogeochemical cycles (PISCES)Laurent Bopp (LSCE) - Patricia Cadule (IPSL)
Current and future climate changes
Jean-Louis Dufresne(LMD) - Olivier Boucher (LMD)
Paleoclimate and last millennium
Pascale Braconnot - Masa Kageyama (LSCE)
“Near-term” prediction (seasonal to decadal)
Eric Guilyardi (LOCEAN) - Juliette Mignot (LOCEAN)
Evaluation of the models, present-day and future climate change analysis
Sandrine Bony (LMD) - Patricia Cadule (IPSL) - Marion Marchand (LATMOS) - Juliette Mignot
(LOCEAN) – Jérôme Servonnat (LSCE)
Regional climatesRobert Vautard (LSCE), Laurent Li (LMD)Atmospheric chemistry and aerosols (INCA,
INCA_aer, Reprobus)Anne Cozic (LSCE) - M. Marchand (LATMOS)
Continental processes (ORCHIDEE)Philippe Peylin (LSCE) - Josefine Ghattas (IPSL)
ICMC organisationPI: J-L Dufresne; Office: L. Bopp, MA Foujols, J. Mignot
Steering committee
![Page 5: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/5.jpg)
Outline
• IPSL climate modelling centre (ICMC) presentation
• IPSLCM history and perspective • Mini how to use modipsl/libIGCM• Post-processing with libIGCM• Monitoring a simulation• Hands-on
![Page 6: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/6.jpg)
IPSLCM history
![Page 7: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/7.jpg)
IPSLCM history and scientific articles
1990 1995 2001 2007 2013
FAR SAR TAR AR4 AR5 IPCC reports
CMIP projectsCMIP 1 & 2 CMIP3
CMIP5
IPSL-CM1
IPSL-CM2
IPSL-CM4
IPSL-CM5
10+ articles
some articles
few articles
30+ articles
IPSL-CM6
![Page 8: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/8.jpg)
LMDZ : atmospheric componenthttp://lmdz.lmd.jussieu.fr/?set_language=en
Next LMDZ training session : 9-11 December 2013inscription before 15th November http://studs.unistra.fr/studs.php?sondage=1wgk8t9v44nsml27
![Page 9: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/9.jpg)
Introduction to LMDZ
![Page 10: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/10.jpg)
NEMO: oceanic componenthttp://www.nemo-ocean.eu
![Page 11: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/11.jpg)
Short history of IPSL modelhttp://icmc.ipsl.fr/index.php/icmc-models
![Page 12: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/12.jpg)
1979 : 1st Linpack performance list
80 Mflops
![Page 13: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/13.jpg)
Supercomputers timeline : top500.org
*10/4 years
![Page 14: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/14.jpg)
Complexity and resolution of models
IPCC, AR4, WG1, Chap. 1, fig 1.2 and 1.4
![Page 15: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/15.jpg)
top500.org : number of CPUS/cores
1 000
100 000
10
1993 2003 2013
![Page 16: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/16.jpg)
Technical challenges : HPC
• More parallelism in component :– MPI : messages programming – hybrid ie MPI/OpenMP : directives and shared
memory
• More parallelism in coupled model– 3 executables at least – each with MPI or MPI/OpenMP– more executables with XIOS : IO servers
• Huge amount of data produced, to be analysed
![Page 17: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/17.jpg)
on the road for IPSL-CM6• New physical package : LMDZ, NEMO, ORCHIDEE• Increased H and V resolutions• Ensembles of simulations• Longer simulations : paleo• More complexity : INCA chemistry added• More processors used in parallel• New dynamical core : DYNAMICO• Optimisation in IO• Improvement and Reliability of libIGCM
![Page 18: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/18.jpg)
Outline
• IPSL climate modelling centre (ICMC) presentation
• IPSLCM history and perspective • Mini how to use modipsl/libIGCM• Post-processing with libIGCM• Monitoring a simulation• Hands-on
![Page 19: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/19.jpg)
Récupérer, compiler et lancer une configuration de type _v5
1. Accès à MODIPSL svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl
2. Accès à IPSLCM5_v5cd modipsl/util ; ./model IPSLCM5_v5
3. Installation des Makefilescd modipsl/util ; ./ins_make
4. Compilation cd modipsl/config/IPSLCM5_v5 ; gmake + resolution choisie
5. Installation de l’expérience type (et post-traitements) cp EXPERIMENT/IPSLCM5/piControl/config.card . vi config.card ### JobName=MYEXP ../../util/ins_job ### recopie repertoire piControl dans MYEXP
avec COMP, DRIVER, PARAM • Soumission du Job de lancement
cd modipsl/config/IPSLCM5_v5/MYEXP; ccc_msub Job_MYEXPllsumbmit Job_MYEXP
![Page 20: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/20.jpg)
Com
putin
gsources of components
IPSL cvs/svn servers
LibIGCM
Specific configuration dowloading
Compilation
Job set up and submission
Connection
Fron
t End
Simulation set up
Modipsl
LibIGCM Physical package choice and set up
![Page 21: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/21.jpg)
Generical job: AA_Job
PeriodLength
![Page 22: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/22.jpg)
libIGCM library : schematic descriptionEXP00 EXP00/DRIVER
driver
card
EXP00/COMP
![Page 23: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/23.jpg)
RebuildFrequency
PackFrequency
SeasonalFrequency
TimeSeriesFrequency
Com
putin
g jo
b
Post
-pro
cess
ing
jobs
PackFrequency
![Page 24: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/24.jpg)
TGCC computers and file system in a nutshell
curiehybrid nodes
-q hybrid
curiehybrid nodes
-q hybrid
curiethin nodes-q standard
curiethin nodes-q standard
curielarge nodes
-q xlarge
curielarge nodes
-q xlarge
dods/storedods/store
$HOME
$CCCSTOREDIR
$CCCWORKDIR
$SCRATCHDIR
HPSS : Robotic tapes
curiefront-end
curiefront-end
Computers
sourcessmall results IGCM_OUT :
MONITORING/ATLAS
temporary REBUILDIGCM_OUT :
files to be packedoutputs of post-proc
jobs
IGCM_OUT : Packed results
Output, Analyse SE and TS
Small precious filesSaved space
File system
dods_cp
cp
ccc_hsm get
airainfront-end
airainfront-end
airainnodesairainnodes
cpdods/workdods/workdods_cp
October 2013Temporary
spaceSaved space
Non saved space
Space on tapes
computecompute
loginlogin
Visible from www
quotasquotas
![Page 25: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/25.jpg)
Job_EXP00Job_EXP00
Com
pute
curie
Job_EXP00Job_EXP00 Job_EXP00Job_EXP00
TGCC PeriodLength PeriodLength
$SCRATCHDIR/IGCM_OUT/.../REBUILD
$SCRATCHDIR/IGCM_OUT/XXX/Restart Debug
DodsCopy=TRUE/FALSE
ncrcat
PackFrequency
$CCCSTOREDIR/IGCM_OUT/XXX/Output
pack_outputpack_output
PackFrequency
$CCCSTOREDIR/IGCM_OUT/.../RESTART DEBUG
Post
curietarpack_restart
pack_debugpack_restartpack_debug
create_tscreate_ts
curiemonitoringmonitoring
Post
TimeSeriesFrequency
TS et SE : $CCCSTOREDIR/IGCM_OUT/… dods/storeMONITORING et ATLAS : $CCCWORKDIR dods/work
create_secreate_se
SeasonalFrequency
atlasatlas
Post
RebuildFrequency
rebuildrebuild
$SCRATCHDIR/IGCM_OUT/XXX/Output
curie
![Page 26: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/26.jpg)
IDRIS computers and file system in a nutshell
dodsdods
$HOME
$HOME
$WORKDIR $WORKDIR
Robotic tapesIGCM_OUT :
Output, AnalyseMONITORING/
ATLAS
$HOME
$TMPDIR
sourcessmall results
temporary REBUILDIGCM_OUT :
files to be packedoutputs of post-proc
jobs
gayagaya
mfput/mfget
dods_cp
mfput/mfget
dmput/dmget
adappcomputeadapp
computeada
computeada
computeadapp
front-endadapp
front-endturing
front-endturing
front-endturingcalculturingcalcul
$TMPDIR $TMPDIR
October 2013Temporary
spaceSaved space
Non saved space
Space on tapes
Visible from www
File system
computecompute
loginlogin
Small precious filesSaved space
![Page 27: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/27.jpg)
Job_EXP00Job_EXP00
Com
pute
ada
Job_EXP00Job_EXP00 Job_EXP00Job_EXP00
IDRIS PeriodLength PeriodLength
$WORKDIR/IGCM_OUT/.../REBUILD
$WORKDIR/IGCM_OUT/XXX/Restart Debug
DodsCopy=TRUE/FALSE
ncrcat
PackFrequency
gaya:IGCM_OUT/XXX/Output
pack_outputpack_output
PackFrequency
gaya:IGCM_OUT/.../RESTART DEBUG
Post
adapptarpack_restart
pack_debugpack_restartpack_debug
create_tscreate_ts
adappmonitoringmonitoring
Post
TimeSeriesFrequency
gaya:IGCM_OUT/… dods.idris.fr
create_secreate_se
SeasonalFrequency
atlasatlas
Post
RebuildFrequency
rebuildrebuild
$WORKDIR/IGCM_OUT/XXX/Output
adapp
![Page 28: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/28.jpg)
Outline
• IPSL climate modelling centre (ICMC) presentation
• IPSLCM history and perspective • Mini how to use modipsl/libIGCM• Post-processing with libIGCM• Monitoring a simulation• Hands-on
![Page 29: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/29.jpg)
Time Series : create_ts.job• A Time Series is a file which contains a single variable over the whole simulation period
(ChunckJob2D = NONE) or for a shorter period for 2D (ChunckJob2D = 100Y) or 3D (ChunckJob3D = 50Y) variables.
• The write frequency is defined in the config.card file: TimeSeriesFrequency=10Y indicates that the time series will be written every 10 years and for 10-year periods.
• The Time Series are set in the COMP/*.card files by the TimeSeriesVars2D and TimeSeriesVars3D options.
• The Time Series coming from monthly (or daily) output files are stored on the file server in the IGCM_OUT/TagName/[SpaceName]/[ExperimentName]/JobName/Composante/Analyse/TS_MO and TS_DA directories.
• Bonus : TS_MO_YE (for annual mean time series) are produced for all TS_MO variables• You can add or remove variables to the TimeSeries lists according to your needs.
[OutputFiles]List= (histmth.nc, ${R_OUT_ATM_O_M}/${PREFIX}_1M_histmth.nc, Post_1M_histmth),\...[Post_1M_histmth]Patches= ()GatherWithInternal = (lon, lat, presnivs, time_counter, time_counter_bnds, aire)TimeSeriesVars2D = (bils, cldh, ......ChunckJob2D = NONETimeSeriesVars3D = (upwd, lwcon, ......ChunckJob3D = OFF
[Post]...#D- If you want to produce time series, this flag determines#D- frequency of post-processing submission (NONE if you don't want)TimeSeriesFrequency=10Y
config.card
COMP/lmdz.card
![Page 30: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/30.jpg)
MONITORING : dods
![Page 31: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/31.jpg)
Intermonitoring : http://webservices.ipsl.jussieu.fr/monitoring/
![Page 32: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/32.jpg)
How to add a new variable in MONITORING
• You can add or change the variables to be monitored by editing the configuration files of the monitoring. Those files are defined by default for each component.
• The monitoring is defined here: ~compte_commun/atlas For example for LMDZ on curie : ~p86ipsl/monitoring01_lmdz_LMD9695.cfgFor example for LMDZ on adapp : ~rpsl035/monitoring01_lmdz_LMD9695.cfg
• You can change the monitoring by creating a POST directory which is part of your configuration. Copy a .cfg file and change it the way you want.
• use ferret language• You can monitor variables produced in time series and stored in TS_MO
#--------------------------------------------------------------------------------------------------------# field | files patterns | files additionnal | operations | title | units | calcul of area#-------------------------------------------------------------------------------------------------------- nettop_global | "tops topl" | LMDZ4.0_9695_grid.nc | "(tops[d=1]-topl[d=2])" | "TOA.
total heat flux (GLOBAL)" | "W/m^2" | "aire[d=3]"
POST/monitoring01_lmdz_LMD9695.cfg
![Page 33: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/33.jpg)
Seasonal mean : create_se.job• A seasonal means files (SE) contain averages for each month of the year (jan, feb,...) for a
frequency defined in the config.card files• SeasonalFrequency=10Y The seasonal means will be computed every 10 years.• SeasonalFrequencyOffset=0 The number of years to be skipped for calculating seasonal means.• All files with a requested Post (Seasonal=ON in COMP/*card) are then averaged within the ncra
script before being stored in the directory: • IGCM_OUT/IPSLCM5A/DEVT/pdControl/MyExp/ATM/Analyse/SE. There is one file per
SeasonalFrequency=10Y• ATLAS are launched by create_se. ATLAS sources are : ~rpsl035 ~p86ipsl/atlas
[OutputFiles]List=(histmth.nc, ${R_OUT_ATM_O_M}/${PREFIX}_1M_histmth.nc, Post_1M_histmth),\...[Post_1M_histmth]...Seasonal=ON
#========================================================================#D-- Post -[Post]...#D- If you want to produce seasonal average, this flag determines#D- the period of this average (NONE if you don't want)SeasonalFrequency=10Y#D- Offset for seasonal average first start dates ; same unit as SeasonalFrequency#D- Usefull if you do not want to consider the first X simulation's yearsSeasonalFrequencyOffset=0
config.card
COMP/lmdz.card
![Page 34: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/34.jpg)
Outline
• IPSL climate modelling centre (ICMC) presentation
• IPSLCM history and perspective • Mini how to use modipsl/libIGCM• Post-processing with libIGCM• Monitoring a simulation• Hands-on
![Page 35: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/35.jpg)
Monitoring the simulation
Verification and Correction
![Page 36: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/36.jpg)
Monitoring a simulation• We strongly encourage you to check your
simulation frequently during run time. First of all, check job status :ccc_mstatllq
• Real time limit exceeded : jobs are killed without any message on ada
• RunChecker.job : This tool, provided with libIGCM, allows you to find out your simulations' status.
• One historical simulation, 156 years : 1850-2005 is composed by 50 computing jobs and 1000 post-processing jobs
Documentationhttp://forge.ipsl.jussieu.fr/igcmg/wiki/
platform/en/documentation
![Page 37: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/37.jpg)
Monitoring a simulation : mail• You receive a message at the end of the simulation• The simulation could be completed or failed
Début du message réexpédié :
De : [email protected]
Objet : MyJobTest failedDate : 22 octobre 2013 17:17:41 UTC+02:00À : [email protected]
Dear rpsl003,
De : [email protected]
Objet : COURSNIV2 completedDate : 22 octobre 2013 18:29:24 UTC+02:00À : [email protected]
Dear rpsl003,
Simulation COURSNIV2 completed on supercomputer ada027Simulation started : 20000101Simulation ended : 20000102
Output files are available in /u/rech/psl/rpsl003/IGCM_OUT/IPSLCM5A/DEVT/pdControl/COURSNIV2Files to be rebuild are temporarily available in /workgpfs/rech/psl/rpsl003/IGCM_OUT/IPSLCM5A/DEVT/pdControl/COURSNIV2/REBUILDPre-packed files are temporarily available in /workgpfs/rech/psl/rpsl003/IGCM_OUT/IPSLCM5A/DEVT/pdControl/COURSNIV2Script files, Script Outputs and Debug files (if necessary) are available in
/gpfs5r/workgpfs/rech/psl/rpsl003/ADA/COURS/NIV2/IPSLCM5_v5/modipsl/config/IPSLCM5_v5/COURSNIV2Greetings!
Check this out for more information : https://forge.ipsl.jussieu.fr/igcmg/wiki/platform/documentation
![Page 38: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/38.jpg)
Monitoring a simulation : run.card• When the simulation has started, the file run.card is created by libIGCM using the
template run.card.init. • run.card contains information of the current run period and the previous periods
already finished. • This file is updated at each run period by libIGCM. • You can find here information of the time consumption of each period.• The status of the job is set to OnQueue, Running, Completed or Fatal.
[Configuration]#last PREFIXOldPrefix= COURSNIV2_20000103#Compute date of loopPeriodDateBegin= 2000-01-04PeriodDateEnd= 2000-01-04CumulPeriod= 4# State of Job "Start", "Running", "OnQueue", "Completed"
PeriodState= CompletedSubmitPath= /gpfs5r/workgpfs/rech/psl/rpsl003/ADA/COURS/NIV2/IPSLCM5_v5/modipsl/config/IPSLCM5_v5/COURSNIV2#========================================================================[PostProcessing]TimeSeriesRunning=nTimeSeriesCompleted=#========================================================================[Log]# Executables SizeLastExeSize= ( 88011086, 0, 0, 19956686, 0, 0, 1523952 )#-----------------------------------------------------------------------------------------------------------------------------------# CumulPeriod | PeriodDateBegin | PeriodDateEnd | RunDateBegin | RunDateEnd | RealCpuTime | UserCpuTime |#-----------------------------------------------------------------------------------------------------------------------------------# 1 | 20000101 | 20000101 | 2013-10-22T17:53:48 | 2013-10-22T17:55:10 | 82.01000 | 4.21000 |# 2 | 20000102 | 20000102 | 2013-10-22T18:28:03 | 2013-10-22T18:29:17 | 74.19000 | 4.09000 |# 3 | 20000103 | 20000103 | 2013-10-23T17:28:50 | 2013-10-23T17:30:26 | 95.21000 | 4.30000 |
run.card
![Page 39: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/39.jpg)
Verification and correction 1/6• Where did the problem occur ?
• 1 "failed" email : Main computation job
=> gaya stopped at IDRIS, hardware problem ? Check Script_output_xxxx.
=> When gaya restarted, or if there isn't any clear error message, try relaunching (after a clean_month):
path/to/libIGCM/clean_month.jobccc_msub (llsubmit) Job_...
![Page 40: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/40.jpg)
Verification and correction 2/6• Where did the problem occur ?
• 1 "failed" email : Main computation job : analyse Script_output_xxxx
######################################## ANOTHER GREAT SIMULATION ######################################## 1ère partie (copying the input files)######################################## DIR BEFORE RUN EXECUTION ######################################## 2ème partie (running the model)######################################## DIR AFTER RUN EXECUTION ######################################## 3ème partie (post-processing)#######################################
http://forge.ipsl.jussieu.fr/igcmg/wiki/platform/en/documentation/G_suivi#AnalyzingtheJoboutput:Script_Output
![Page 41: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/41.jpg)
Verification and correction 3/6--> analyse Script_output_xxxx :In general, if your simulation stops you can look for the keyword "IGCM_debug_CallStack" in this file.This keyword will come after a line explaining the error you are experiencing.
=====================================================================EXECUTION of : mpirun -f ./run_file > out_run_file 2>&1Return code of executable : 1IGCM_debug_Exit : EXECUTABLE
!!!!!!!!!!!!!!!!!!!!!!!!!!!! IGCM_debug_CallStack !!!------------------------!
!------------------------!IGCM_sys_Cp : out_run_file xxxxxxxxxxxx_out_run_file_error=====================================================================
![Page 42: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/42.jpg)
Verification and correction 4/6--> Check closely the sub directory Debug (if it exists)
Check file xxxxx_error in Debug/ : – contains LMDZ standard output. LMDZ often fails in hgardfou.
Stopping in hgardfou– contains abends (abnormal termination / exception) of each and every
component.
Check standard outputs for NEMO, ORCHIDEE, INCA, OASIS– Debug/xxxx_ocean.output– Debug/xxxx_output_orchidee– Debug/xxxx_inca.out– Debug/xxxx_cplout
![Page 43: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/43.jpg)
RunChecker.job• RunChecker.job helps you to monitor all the jobs produced by libIGCM for a simulation
![Page 44: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/44.jpg)
RunChecker.job : usage and options
This script can be launched from anywhere.
Usage:path/to/libIGCM/RunChecker.job [-u user] [-q] [-j n] [-s] job_namepath/to/libIGCM/RunChecker.job [-u user] [-q] [-j n] -p config.card_pathpath/to/libIGCM/RunChecker.job [-u user] [-q] [-j n] -r
Options :-h : print this help and exit-u user : owner of the job-q : quiet-j n : print n post-processing jobs (default is 20)-s : search for a new job in $WORKDIR and fill in the catalog before printing
information-p path : give the absolute path to the directory containing the config.card
instead of the job name (needed only once)-r : check all running simulations.
1) path/to/libIGCM/RunCkecker.job –p $CCCWORKDIR/CURIE/CMIP5/R1414/IPSLCM5A_20120731/modipsl/config/IPSLCM5A/v5.rcp45CMR2
2) path/to/libIGCM/RunCkecker.job v5.rcp45CMR2
![Page 45: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/45.jpg)
![Page 46: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/46.jpg)
Pb !
STOP (Fatal into run.card)
![Page 47: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/47.jpg)
Verification and correction 5/6• You have received 2 "failed" emails or
RunChecker status is abnormal ie : red• Analyse the situation:
– Simple case:• Re-submit rebuild, pack_debug or
pack_restart jobs• Re-submit pack_output
– Less simple case:• Use clean_year to go back to a
healthy situation• Holes in the data
path/to/libIGCM/clean_year.job [SSAA]• all data from current year to SSAA
(included) will be deleted.• Restart the simulation
https://forge.ipsl.jussieu.fr/igcmg/wiki/platform/en/documentation/G_suivi#Startorrestartpostprocessingjobs1
![Page 48: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/48.jpg)
TimeSeries_Checker.job• Install a dedicated directory• Copy required files and directories : config.card, run.card, COMP, POST• Copy from libIGCM the script : TimeSeries_Checker.job• Modify the job : libIGCM, name of the simulation, ...• Look at the documentation :
https://forge.ipsl.jussieu.fr/igcmg/wiki/platform/en/documentation/G_suivi#TimeSeries_checker.job-Recommendedmethod
> mkdir POST_REDO> cd POST_REDO> cp –pr COMP POST config.card run.card .> cp ../../../../libIGCM/TimeSeries_Checker.job .> vi TimeSeries_Checker.job# Check/Modify : libIGCM= SpaceName= ExperimentName= JobName=
CARD_DIR= BRIDGE_MSUB_PROJECT=gen2211> ./TimeSeries_Checker.job Answer y to submit create_ts.job
ksh > ./TimeSeries_Checker.job 2>&1|tee TSC_OUT_TO_KEEP
![Page 49: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/49.jpg)
Verification and correction 6/6• Everything went ok :
– End of simulation email– No anomaly detected by
RunChecker
• TimeSeriesChecker (and SE_checker):Checks existing time series et submit create_ts jobs to build the missing ones
• Keep in mind:– Rebuild jobs automatically
submit pack jobs, as well as corresponding TS and SE.
– Pack, TS and SE jobs may be re-submitted independently from a rebuild job
![Page 50: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/50.jpg)
The END! (so soon?)
[email protected] list to ask for help and to share
information with other users [email protected]
![Page 51: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/51.jpg)
How to redo a rebuild?• Install a dedicated directory• Copy required files and directories : config.card, run.card, COMP, POST• Copy from libIGCM job to be relaunched : rebuild...• Modify the job : dates, name of the simulation, ...• Look at the documentation :
https://forge.ipsl.jussieu.fr/igcmg/wiki/platform/en/documentation/G_suivi#Startorrestartpostprocessingjobs1
> mkdir POST_REDO> cd POST_REDO> cp –pr COMP POST config.card run.card .> cp ../../../../libIGCM/rebuild_fromWorkdir.job .> vi rebuild_fromWorkdir.job# Check/Modify : libIGCM= PeriodDateBegin= NbRebuildDir=
REBUILD_DIR=> llsubmit rebuild_fromWorkdir.job> ccc_msub rebuild_fromWorkdir.job
![Page 52: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/52.jpg)
New in libIGCM v2 : Accounting mail• You receive a message after the first job with some information regarding
time used by the simulation• Take it into account before launching a simulation, especially large
simulations
De : [email protected]
Objet : MyJobTest AccountingDate : 22 octobre 2013 17:23:52 UTC+02:00À : [email protected]
Dear rpsl003,
this mail will be sent once for the simulation MyJobTest you recently submitted
The whole simulation will consume around 3.939110 hours. To be compared with your project allocation.
The recommended PeriodNb for a 24 hours job seems to be around 974.839488. To be compare with the current setting (Job_MyJobTest parameter) : PeriodNb=3
Greetings!
Check this out for more information : https://forge.ipsl.jussieu.fr/igcmg/wiki/platform/documentation
![Page 53: Training session 2 : Advanced training course on modipsl and libIGCM November 14 th 2013, MdS](https://reader035.vdocument.in/reader035/viewer/2022062409/56814c4d550346895db956bd/html5/thumbnails/53.jpg)
More information
• Audit report on IPSL workflow used at TGCC and IDRIS
• IPSL wiki and documentation : https://forge.ipsl.jussieu.fr/igcmg/wiki/platform/en/documentation