introduction to the standard compile environment (sce) of the

22-24 Feb 05 Workshop DKRZ / M&D Stephanie Legutke

Introductionto the

Standard Compile Environment (SCE)of the

Integrated Model & Data Infrastructure (IMDI)

Stephanie Legutke, MPI-M M&D


Background

PRISM (Dec 2001-Nov 2004): Develop for the European Earth System (ES) modelling community a software infrastructure to set up compile run analyse coupled ES model experiments

and thereby establish a European ES research network

M&D: Make the SCE and SRE developed in PRISM become part of the IMDI


Requirements ...

… to the SCE infrastructure software

Integrate any ES climate research model …

… on any platform

Allow for easy replacement of components in coupled models (CplM)

Interface to a GUI


Requirements …

Minimum impact on component source code

Give a common look&feel with all models and experiments

Automate but allow for easy customizing

Keep it simple

[Low maintenance]

… from the model developers and users


Components of the SCE

Coupled models, component main & sub models, and libraries

Standards

High level design of the compile system

Compile time configuration of coupled/component models

Component model compile scripts generation

(g)make (file)


OCEAN

Coupled Earth System Models

Sea Ice

ATMOSPHERE

MarineBiogeo-

chemistry

LandSurface

Atmospheric Chemistry

OASIS3


Coupled Earth System Models

OCEAN

ATMOSPHERE

MarineBiogeo-

chemistry

LandSurface

Atmospheric Chemistry

AnotherSea Ice

OASIS3

Main models

Sub models

Message passing

SBR parameter list


SCE standards for ....

Package rule for main&sub model exchange: No use of other component‘s data blocks ; use message passing or SBR param. lists instead

OASIS3 software if different executables:via OASIS executable or directly using PSMILe(MPI) calls(PSMILe=PRISM System ModelI Interface Library)

Configuration for coupling with partner models by conditional compiling (cpp flag ‘__cpl_partnermodel‘)

... coupling


SCE standards for ....

... component model codes Components have to be independent from each other

(for flexible component exchange)

Avoid unresolved references by cpp flag wraping Standard directory structure (to enable use of simple tools)

... source code Single F90 MODULE per file File name = MODULE name Suffixes: F90, f90, F, f, c Single occurence of each basename Non portable code wraped with cpp flags


USE, CALL, INCLUDE

SCE: Source code directory

ok

src2

src1

src

m od 1

s rc

su p p ort

s rc

p sm ile

s rc

m od -lib

lib

s rc

m od 2

in c lu d e

s rc s rc2 in c lu d e

m od 1

m od

s rc

S R C R O O T

ok for call of SBR if mod2 is submodel of mod1


SCE: Source code reorganisation

p ack1

p ack2

s rc m od

in c lu d e

s rc

m od

p ackag e in c lu d e

S R C R O O T

befores rc_ p ack1 s rc_ p ack2

m od

in c lu d e

p ackag e_ s rc

in c lu d e p ackag e_ m od

m o d e l

m od

s rc

S R C R O O T

after


COMP_mod1_ID.nodeUser

lib*.a

callCOMP_libs.node ”list2”

return

Model+submodel compilation

mod1_mod2_ID[.MPIn].x MPIn=MPI or MPI2

COMP_libs.node ”list1”

callmod2 / libmod2_ID[.MPIn].a

COMP_mod2_ID.nodecall

return

return

mod1 / libmod1_mod2 _ID [.MPIn].a


Build directory (libraries and executables)

m od1_m od2_ID _[MPI[1,2]].x

bin build

lib lib1.a...lib libn.a libm od1_m od2_ID _...a

m od1

libm od2_ID _[MPI[1,2]].a

m od2

m odn

lib

a rch

m od1

...

m odn

m od

lib1

...

libn

lib

src

SR C R O O T


COMP_mod1_ID.node

Component model compilation

mod1_mod2_ID[.MPIn].x

lib*.amod2 / libmod2_ID[.MPIn].amod1 / libmod1_mod2 _ID [.MPIn].a

lib*.amod2 / libmod2_ID[.MPIn].amod1 / libmod1_mod2 _ID [.MPIn].a


COMP_mod2_ID.node


call

call

call

return

return

return

User

MPIn=MPI or MPI2


COMP_libs.node “list”

COMP_mod1_ID.node all / lib / clean / tar

“list”

User

list of libraries:

make src / Makefile

*.o / *.mod*.o / *.mod

Model compilation

Top_Makefilemake

create (in-doc.)

mod1_ID[_MPIn].x

src directories:

make Makefile

lib*.alib*.a

Top_Makefilemake

create (in-doc.)


Build directory: *.o/*.mod

bin

m odn_ID

m od

libn

lib

build

lib

a rch

m od1

...

m odn

m od

lib1

...

libn

lib

src

SR C R O O T

*.o*.mod

*.a

modn*_ID.a

*_ID.x

*.o*.mod


Utilities & Tools for compilation

… for the specific platform … for the specific component model … configured for the coupled configuration

No compile scripts are provided …

… but tools that generate them …

… Example how to do that =>


Creating compile scripts

Create_COMP_cpl_mod.ksh cplmod [ID [node]]

Create_COMP_models.frm mod1 MPI[1,2] - - node ID “mod1 mod2 mod3”Create_COMP_models.frm mod2 MPI[1,2] - - node ID “mod1 mod2 mod3”Create_COMP_models.frm mod3 MPI[1,2] - - node ID “mod1 mod2 mod3”Create_COMP_models.frm oasis3 MPI[1,2] - - nodeCreate_COMP_libs.frm MPI[1,2] - - node

COMP_mod1_ID.nodeCOMP_mod2_ID.nodeCOMP_mod3_ID.node COMP_oasis3.node COMP_libs.node

mod1_mod2_ID.MPI[1,2].xmod2_ID.MPI[1,2].amod3_ID.MPI[1,2].x oasis3.MPI[1,2].x

…. , psmile.MPI[1,2].a

cplmod = mod1+mod2 & mod3 & OASIS3:

bold: to be typed by user (scripting) additional configuring by editing


…include GUIspecific_$model.hinclude Sitespecific_$node.hinclude Cppflags_$model.hinclude Compile_mode_$model_$node.hinclude Make_model.h

…

m4 input file

Compile Script Assembly

Create_COMP_models.frm model “” – “” node ID “partner models”

COMP_model_ID.nodem

4

*.h


list of libraries:

make src / Makefile

COMP_libs.node “list”call with libraries “list”

SCE : Model compilation

Top_Makefilemake

create (in-doc.)

mod1_ID[_MPIn].x

src directories:

make Makefile

lib*.alib*.a

Top_Makefilemake

create (in-doc.)

*.o / *.mod*.o / *.mod

COMP_mod1_ID.node all / lib / clean / tar User


SCE: (GNU) Make

All compilation is based on the (g)make software:Make ‘targets’ having ‘prerequisites’ with well defined ‘rules’ while avoiding redundant actions:

Make –f Makefile

…Target: prerequisite1, prerequisite2 ….

rule …

Targets, rules & prerequisites are defined in Makefile:


(GNU) Make targets

Executables: OASIS3.x, ECHAM5.x, *.x

Libraries: *.a

Model libraries: *_ID.a

Binary object code: *.o

FORTRAN90 Modules: *.mod


(GNU) Portable Makefiles

All non portable code exported from the compile scripts

1 Makefile for each source directory with compiler input

All CMs Makefiles have the same look and feel

Exception: prerequisites Tool for generation of *.o prerequisites

All library Makefiles have the same look and feel


Example Makefile: Def. of variables

SRCS1 = $(shell find ../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)) –name '*.F90' -print)

OBJS1 = $(patsubst ../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/%.F90, %.o, $(SRCS1))

….

SRCS5 = $(shell find ../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)) -name '*.c' -print)

OBJS5 = $(patsubst ../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/%.c, %.o, $(SRCS5))

VPATH = .:../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)):\ ../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/include:\

../../../../src/mod/$(MODEL_DIR)/include:\ ../../lib/psmile.${CHAN}:\ $(NETCDF_INCLUDE):$(MPI_INCLUDE):$(SYS_INCLUDE):\ ../../../lib/$(MODEL_DIR):\ ../../../lib/$(SUBMOD1):\ ../../../lib:\ ../../../../src/mod/$(MODEL_DIR)/make_dummies

LIBRARY = ../../../lib/$(MODEL_DIR)/lib$(MODLIB).a

PROG = ../../../bin/$(EXEC).x

INCLS = -I../../../../src/mod/$(MODEL_DIR)/include \ -${I4mods}../../lib/clim.${CHAN}

INCLSC = -I../../../../src/mod/$(MODEL_DIR)/include


Example Makefile: Targets & Rules

clean: rm -f $(LIBRARY) i.* *.o *.mod Top_Makefile_*

all: $(PROG)

lib: $(LIBRARY)

$(PROG): $(DEPLIBS) $(LIBRARY) $(F90) $(LDFLAGS) -o $@ $(MAINPRG).o $(LIBS)

$(LIBRARY): $(OBJS1) $(OBJS2) $(OBJS3) $(OBJS4) $(OBJS5) $(AR) $(ARFLAGS) $(LIBRARY) *.o

.SUFFIXES:

.SUFFIXES: .o .c .f .F .f90 .F90

%.o: %.F90 $(F90) $(F90FLAGS) $(INCLS) -c $<

…%.o: %.f

$(f) $(fFLAGS) $(INCLS) -c $<

%.o: %.c $(CC) $(CCFLAGS) $(INCLSC) -c $<


Generating .o prerequisites

mod_unitncdf.o: mod_kinds_oasis.oalloc_src.o: mod_anais.o mod_analysis.o mod_coast.o mod_experiment.o \

mod_extrapol.o mod_kinds_oasis.o mod_memory.o mod_nproc.o \mod_parallel.o mod_parameter.o mod_pipe.o mod_rainbow.o mod_sipc.o \mod_string.o mod_timestep.o mod_unitncdf.o

dealloc_src.o: mod_anais.o mod_analysis.o mod_coast.o mod_experiment.o \mod_extrapol.o mod_kinds_oasis.o mod_memory.o mod_nproc.o \mod_parallel.o mod_parameter.o mod_pipe.o mod_rainbow.o mod_sipc.o \mod_string.o mod_timestep.o mod_unitncdf.o

extrap.o: mod_extrapol.o mod_kinds_oasis.o mod_parameter.o mod_printing.o \mod_unit.o

getfld.o: mod_analysis.o mod_clim.o mod_experiment.o mod_hardware.o \mod_kinds_oasis.o mod_label.o mod_memory.o mod_parameter.o \mod_printing.o mod_sipc.o mod_string.o mod_timestep.o mod_unit.o \mod_unitncdf.o netcdf.inc

Append_Dependencies model_name src_directory


SCE : Util directory Structure

C rea te_C O MP_libs.frm

"_"_m odels.frm

"_"_cpl_m odels.ksh

include

include_node1

include_...

fra m es

Append_dependencies

com pile C O MP_libs.node running

util

b in

build

lib

a rch

m od1

...

m odn

m od

lib1

...

libn

lib

src

SR C R O O T


Documentation & URLs

HandbookThe PRISM Standard Compilation EnvironmentStephanie Legutke and Veronika GaylerPRISM Report Series, No. 4, 1. Edition, 15 Jan 2005 http://prism.enes.org/Results/Documents/PRISMReports/Report04.pdf

WP3i (Model Assembly) M&D PRISM web site http://prism.dkrz.de/Workpackages/WP3i/Documentationshttp://prism.dkrz.de/Workpackages/WP3i/Presentations

PRISM project web site http://prism.enes.org

PRISM Portal (under development)http://prism.enes.org/Portal

http://prism.enes.org/Results/Documents/PRISMReports/Report04.pdf

http://prism.dkrz.de/Workpackages/WP3i/Documentations

http://prism.dkrz.de/Workpackages/WP3i/Documentations

http://prism.enes.org/

http://prism.enes.org/Portalg

introduction to the standard compile environment (sce) of the

Documents

model developers

partner models

scecoupled models

component source code

coupled es model experiments

es climate research

coupled models cplminterface

mainsub model exchange