John Donnersjohn.donners@surfsara.nl

EasyBuild@SURFsara

Voetnoot

● SURFsara provides computing to researchers for over 40 years● Part of SURF: a cooperation of universities, research institutes and higher education

in the Netherlands.

SURFsara

Voetnoot

● Cartesius national supercomputer, uses RedHat (upgraded from 6 to 7)● Lisa national cluster, uses Debian (upgraded from 7 to 9)● Grid● Spark cluster● Jupyter notebooks● HPC Cloud● Visualization● Data services – archiving, SURFdrive

Infrastructure

Old style

login per software installationCompile by handAdhoc scriptsuse own compiler wrappers to add location of libraries with a RUNPATHmanually write moduleno dependencies or conflicts in most modules, so users can mix and matchmanually update documentation on the website

IssuesUsers end up with incompatible module combinationsLD_LIBRARY_PATH sometimes ends up in modulesDocumentation is an afterthoughtDefault versions rarely updated, not to surprise users

Voetnoot 4 / 17

New style

login per software installationCompile by handAdhoc scriptsuse own compiler wrappers to add location of libraries with a RUNPATHmanually write moduleno dependencies or conflicts in most modules, so users can mix and matchmanually update documentation on the website

IssuesUsers end up with incompatible module combinationsLD_LIBRARY_PATH sometimes ends up in modulesDocumentation is an afterthoughtDefault versions rarely updated, not to surprise users

Voetnoot 5 / 17

@ SURFsara

● Already discussions about EasyBuild 3 years ago● Main trigger to start using EasyBuild was RPATH support● First test was to install OpenFOAM with RPATH using EasyBuild

● After two fixes to the framework, we could install OpenFOAM using Intel compilers with RPATH support

● Started with a minimal set of toolchains foss/2015b and intel/2016b● foss: 2015b (stops working on Debian 9) → 2017b● intel: 2016b → 2018a

Voetnoot 6 / 17

@ SURFsara

● Used module-command was too old..● Tried Lmod, but that was very slow (yes, with cache). Seems due to many .version files.

Not anymore tested newer releases.● Now using modules-tcl and modules.

● The is some postprocessing after EasyBuild finished:

● Changing some permissions● Additional environment variables SURFSARA_LIBRARY_PATH for compiler wrappers● Add compiler wrappers to compilers, MPI-libraries and binutils.

● Additional scripts for users to install software locally.

Voetnoot 7 / 17

@ SURFsara

● Script to change default version with a message one month before and after:

donners@login2:~$ module av qiimeDefault version for module qiime will change from 1.8.0 to 1.9.1 on 12-02-2018----------------------- /hpc/sw/modules/modulefiles/applications ------------qiime/1.8.0(default) qiime/1.9.0 qiime/1.9.1

donners@login2:~$ module av vegasDefault version for module vegas changed from 0.8.27 to 2.01.17 on 23-12-2017---------------- /hpc/sw/modules/modulefiles/applications ---------vegas/0.8.27 vegas/2 vegas/2.01.17(default)

● Script to deprecate modules with a one month notice:● donners@login2:~$ module show SPAdes/3.5.0● -------------------------------------------------------------------● /hpc/sw/modules/modulefiles/tools/SPAdes/3.5.0:●

● module-whatis Setup environment to use SPAdes 3.5.0● prepend-path PATH /sara/sw/SPAdes-3.5.0/bin● Module SPAdes/3.5.0 is deprecated and will be removed after 12-02-2018● For more information, please contact helpdesk@surfsara.nl● -------------------------------------------------------------------●

Voetnoot 8 / 17

What's the issue with LD_LIBRARY_PATH?

[donners@int2 ~]$ /usr/bin/pythonPython 2.7.5 (default, May 3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import ldap>>>

Voetnoot 9 / 17

What's the issue with LD_LIBRARY_PATH?

[donners@int2 ~]$ module show python-------------------------------------------------------------------/hpc/sw/modules/modulefiles/compilers/python/2.7.9:

module-whatisActivate Python 2.7.9prepend-path PATH /hpc/sw/python-2.7.9/binprepend-path LD_LIBRARY_PATH /hpc/sw/python-2.7.9/lib:/hpc/sw/python-2.7.9/lib/python2.7...-------------------------------------------------------------------[donners@int2 ~]$ module load python

Voetnoot 10 / 17

What's the issue with LD_LIBRARY_PATH?

[donners@int2 ~]$ module load python[donners@int2 ~]$ /usr/bin/pythonPython 2.7.9 (default, Mar 9 2015, 11:04:54) [GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import ldapTraceback (most recent call last): File "<stdin>", line 1, in <module>ImportError: No module named ldap>>>

[donners@int2 ~]$ ldd /usr/bin/pythonlinux-vdso.so.1 => (0x00007fffb3fb0000)libpython2.7.so.1.0 => /hpc/sw/python-2.7.9/lib/libpython2.7.so.1.0 (0x00002ab19f3af000)

Voetnoot 11 / 17

Our eb config

Voetnoot 12 / 17

[basic] repositorypath = /hpc/eb/easyconfigs-surfsara robot-paths = %(repositorypath)s:%(DEFAULT_ROBOT_PATHS)s[config] installpath-modules = /hpc/eb/RedHatEnterpriseServer7/modulefiles installpath-software = /hpc/eb/RedHatEnterpriseServer7 modules-tool = EnvironmentModulesTcl module-syntax = Tcl tmpdir = /tmp/%(USER)s buildpath = /tmp/%(USER)sinclude-easyblocks = /hpc/eb/easyblocks-surfsara/trilinos.py,/hpc/eb/easyblock[github] github-org = sara-nl[override] experimental = 1 rpath = 1 filter-env-vars = LD_LIBRARY_PATH set-gid-bit = 1 umask = 022 download-timeout = 60 optarch=Intel:O2 -xAVX -axCORE-AVX2,MIC-AVX512;GCC:O2 -march=sandybridge

Not-so-

● Installing the same easyconfigs after upgrading the OS:● easyconfigs have more system dependencies than thought (often zlib)● Debian 9 use OpenSSL 1.1, problems with: CMake, Python, ..

● It can be difficult to debug when an easyconfig doesn't install

● DBus from EasyBuild interferes with DBus from OS (on RedHat 7)● Better to depend on OS version

● LLVM built with Intel compiler is unchecked, causes crashes

Voetnoot 13 / 17

How-to-

How to neatly manage Python extensions? Can be installed with● Initial Python installation● Bundle EasyBlock● PythonPackage EasyBlock● pip

When to install packages separately, when as a bundle?Bundle/PythonPackage will reinstall already present packages (dependency resolution goes through EB, not pip)

Voetnoot 14 / 17

How-to-

How to properly manage GPU software?Use CUDA-based toolchain (e.g. GCCcuda, gompic, goolfc)?Few EasyConfigs availableGompic/goolfc incompatible with common intel/foss: different MPIPotential solution: reuse EasyConfigs for common GCCcore subtoolchain, --try-toolchain, or recreate (many) EasyConfigs.

Use foss with CUDA-<vn> as suffix+dependency? (like cuDNN)Should there be a foss + CUDA toolchain?User can load foss-based modulesProblem: non-GPU enabled openMPI (only gromacs easyconfig uses this)

Voetnoot 15 / 17

Thanks!