COMPILING, BUILDING, AND INSTALLING PROGRAMS ON

THE CLUSTER

BUILDING COMPUTER PROGRAMS

•  Theprocessofconver-ngahuman-readablefiletoamachine-readablefile.

#include <stdio.h> int main() { prin9(“Hello World!\n”); return 0; }

01010101000010101010010101110101010010100000010111010100101100011101101010101010101010101010001010101010111111001011110111000011101010101001111010101111010101010000110101010101

Cprogram(simpletextfilewri9eninCprogramminglanguage)

Binaryexecutablefile(asetofCPUinstruc-onsencodedin0’sand1’s)

Sophis-catedprograms(e.g.acompiler)areusedtoperformthismul--stepconversion.

HelloWorld!

THE BUILD PROCESS

Notalllanguagesarecompiledlanguages!Theprocesstothe

leIappliestoprogramswri9eninC,C++,andFortran.

Linker

Assembler

Compiler

Preprocessor

Expandedsourcecode

Sourcecode(e.g.Cprogram)

Assemblycode

Objectcode

Binaryexecutable

Externallibraries

Human-readableHigher-levellanguage

Lower-levellanguage

Machine-readable(i.e.canbeexecutedbyCPU)

PREPROCESSOR

•  Expandsorremovesspeciallinesofcodepriortocompila-on.

.

. #include <stdio.h> . .

.

. #define PI 3.1415 . .

.

. #ifndef FOO_H #define FOO_H #include “myHeader.h” void myFunc(int); #endif . .

Includestatements: Definestatements: Headerguards:

•  Copiescontentsofstdio.hintofile.

•  ReplacesallinstancesofPIwithinfilewith3.1415.

•  Preventsexpandingmul-plecopiesofthesameheaderfilebydefiningaunique“macro”foreachheaderfile.

InC,preprocessordirec-vesbeginwiththe#symbolandare

NOTconsideredCcode.

COMPILER

•  Convertsexpandedsourcecodetoassemblycode.

#include <stdio.h> int main() { prin9(“Hello World!\n”); return 0; }

.

. main: .cfi_startproc pushq %rbp .cfi_def_cfa_offset 16 movq %rsp %rbp . .

•  Assembly-levelinstruc-onsarespecifictoaprocessor’sInstruc-onSetArchitecture(ISA).

•  ExampleISAsarex86,x86_64,andARM.MostmachinesinHPCtodaysupportx86_64.

Portabilityisanissuewithcompiledlanguagessinceassemblylanguagecontains

instruc-onsthatarespecifictoaCPU’sarchitecture.

ASSEMBLER AND LINKER

•  Assembler:convertsassemblycodetoobjectcode.

•  Linker:s-chestogetherallobjectfiles(includinganyexternallibraries)intothefinalbinaryexecutablefile.

•  Objectcodeisinabinaryformatbutcannotbeexecutedbyacomputer’sOS.•  ExternallibrariesareoIendistributedassharedobjectfilesthatareobjectcode.

•  Hidesspecificimplementa-onsincethesefilesarenothumanreadable.•  Noneedtoberecompiledforeachapplica-onthatusesthelibrary.•  Storedefficientlyinbinaryformat.

•  Manyapplica-onsoIencontainmul-plesourcefiles,eachofwhichneedtobeincludedinthefinalexecutablebinary.

•  Thejobofthelinkeristocombinealltheseobjectfilestogetherintoafinalexecutablebinary(a.k.a.“executable”or“binary”)thatcanberun.

Executable

ObjectFile1

ObjectFile2

ExtLibA

USING COMPILERS ON THE CLUSTER (1/3)

IMPORTANTNOTE:Inprac-ce,thestepsperformedbythepreprocessor,compiler,assembler,andlinkeraregenerallyobscuredfromtheuserintoasinglestepusing(inLinux)asinglecommand.Inthenextseveralslides,wewillrefertothissinglecommandasacompiler,butnotethatwe’reactuallytalkingaboutatoolthatisapreprocessor+compiler+assembler+linker.

•  GCC:GNUCompilerCollec<on•  Freeandopensource•  MostwidelyusedsetofcompilersinLinux•  Ccompiler:gcc•  C++compiler:g++•  Fortrancompiler:gfortran

•  IntelCompilerSuite•  Licensedandclosedsource,butACCRE

purchasesalicense•  OIenproducesfasterbinariesthanGCC•  Occasionallymoredifficulttobuildcode

duetolackofcommunitytes-ng•  Ccompiler:icc•  C++compiler:icpc•  Fortrancompiler:ifort

USING COMPILERS ON THE CLUSTER (2/3)

gcchello.c

•  BuildsCprogramwiththeGCCCcompiler.•  Producesabinarycalleda.outthatcanberunbytyping./a.out

gcc–ohellohello.c

•  Producesabinarycalledhellothatcanberunbytyping./hello

Errormessagesresultwhenthebuildprocessfails.Thecompilershouldprovide

detailsaboutwhythebuildfailed.

Warningmessagesoccurwhenaprogram’ssyntaxisnot100%cleartothecompiler,butitmakesanassump-on

andcon-nuesthebuildprocess.

USING COMPILERS ON THE CLUSTER (3/3)

gcc–ohello-Wallhello.c

•  -Wallwillshowallwarningmessages

gcc–ohello-ghello.c

•  -gwillbuildthebinarywithdebugsymbols

gcc–Ehello.c

•  Showexpandedsourcecode

gcc–Shello.c

•  Createassemblyfilecalledhello.s

gcc–chello.c

•  Createobjectfilecalledhello.o

gcc–ohello–O3hello.c

•  -O3willbuildthebinarywithlevel3op-miza-ons•  Levels0to3(mostaggressive)available•  Canleadtofasterexecu-on-mes•  Defaultis–O0inGCCand–O2inIntelsuite

Vectorizedloopexecu-onisenabledwith–O3forGCCand–O2forIntel.

icc–ohello–xHosthello.c•  UseIntel’sCcompilertoaggressivelyop-mizeforthespecificCPU

microarchitecture

Usingthe–xHostop-onleadstopoorbinaryportability.Onlyusethisop-onifyouaresurethebinarywillalwaysbeexecutedonaspecificprocessortype.

EXTERNAL LIBRARIES (1/2)

•  Sta<callyLinkedLibrary:namingconven-on:liblibraryname.a(e.g.libcurl.aisasta-ccurllibrary)

•  DynamicallyLinkedLibrary:namingconven-on:liblibraryname.so(e.g.libcurl.soisadynamiccurllibrary)

•  Linkercopiesalllibraryrou-nesintothefinalexecutable.•  Requiresmorememoryanddiskspacethandynamiclinking.•  Moreportablebecausethelibrarydoesnotneedtobeavailableatrun-me.

•  Onlythenameofthelibrarycopiedintothefinalexecutable,notanyactualcode.•  Atrun-me,theexecutablesearchestheLD_LIBRARY_PATHandstandardpathforthelibrary.•  Requireslessmemoryanddiskspace;mul-plebinariescansharethesamedynamicallylinkedlibraryatonce.•  Bydefault,alinkerlooksforadynamiclibraryratherthanasta-cone.

•  DoNOTneedtospecifytheloca<onofalibraryatbuild<meifit’sinastandardloca<on(/lib64,/usr/lib64,/lib,/usr/lib).Forexample,libc.solivesin/lib64.

EXTERNAL LIBRARIES (2/2)

•  Linkingtolibrariesinnon-standardloca<onsrequiresthefollowinginforma<onatbuild-<me:

•  Nameoflibrary(specifiedwith–llibrarynameflag)•  Loca-onoflibrary(specifiedwith–L/path/to/non/standard/loca-on/lib)•  Loca-onofheaderfiles(specifiedwith–I/path/to/non/standard/loca-on/include)

gcc–L/usr/local/gsl/latest/x86_64/gcc46/nonet/lib–I/usr/local/gsl/latest/x86_64/gcc46/nonet/include–lgsl–lgslcblasbessel.c–Wall–O3–ocalc_bessel

•  Inthisexample,twolibraries(gslandgslcblas)arelinkedtothefinalexecutable.•  Alterna-vely,useLIBRARY_PATHandC_INCLUDE_PATHtospecifyloca-onsoflibrariesandheaders.

•  ChecktheLD_LIBRARY_PATHandoutputofthelddcommandbeforerunningtheprogram:

•  LD_LIBRARY_PATHshowslistofdirectoriesthatlinkersearchesfordynamicallylinkedlibraries•  Runldd./my_progtoseethedynamicallylinkedlibrariesneededbyanexecutableandthecurrentpath

toeachlibrary

PORTABILITY

Supportforspecificvectoriza-onextensionsisalsorequiredfor

portability.Forexample,youcannotbuildaprogramwithAVX2onplaoormAandrunitonplaoormBifAVX2isnot

supportedbyplaoormB!

CanIbuildanexecutableoncomputerAandrunitoncomputerB?

•  CPUinstruc-onsetarchitecture(e.g.x86_64)•  Opera-ngsystem•  Externallibraries

Itdepends!Arethepla/ormsthesame?

Pla/orm

•  ThisiswhyyouoIenseedifferentinstallersfordifferentopera-ngsystems–theinstallerissimplycopyingapre-builtbinarytoyourmachine!

•  DifferentCPUarchitecturesarepresentonthecluster,sobesuretocompilewithoutoverlyaggressiveop-miza-onsorspecifythetargetCPUarchitecture/familyinyourSLURMscript(e.g.#SBATCH--constrain=haswell)

OTHER COMPILER FUN FACTS

•  Manydifferentcompilersexistbutnotallcompilersarecreatedequal!

•  Performanceofprogramcanbeverycompiler-dependent!

•  GCC,Intel,AbsoI,PortlandGroup(PGI),MicrosoIVisualStudio(MSVS),tonameafew.•  Somearefree,othersarenot!•  Itisnotunusual(especiallywithlargeprojects)forcompilerAtobuildaprogramwhilecompilerBfails.•  Errormessagesandlevelsofverbositycanalsovarywidely.

•  Thisisespeciallytrueinscien-ficandhigh-performancecompu-nginvolvingalotofnumericalprocessing.•  Compilerop-miza-onsareespeciallytricky,some-mesthecompilerneedshelpfromtheprogrammer(e.g.

re-factoringcodesothecompilercanmakeeasier/saferdecisionsaboutwhentoop-mizecode).•  Somecompilers(especiallyIntel’s)tendtooutperformtheircounterpartsbecausetheyhavemorein-mate/

nuancedinforma-onaboutaCPU’sarchitecture(whichareoIenIntel-based!).

AUTOMATING THE PROCESS: MAKEFILES (1/3)

•  DefaultfileiscalledMakefileormakefile.•  Allowsbuildprocesstobebrokenupintodiscreetsteps,ifdesired.Forexample,separaterulescanbe

definedfor(i)compiling+assembling,(ii)linking,(iii)tes-ng,and(iv)installingcode.•  Makeanalyzesthe-mestampsofatargetandthattarget’sdependenciestodecidewhethertoexecutea

•  TheMaketoolallowsaprogrammertodefinethedependenciesbetweensetsoffilesinprogrammingproject,andsetsofrulesforhowto(mosto_en)buildtheproject.

Automating the build process

•  make)u2lity))–  Provides)a)way)for)separate)compila2on))–  Describe)the)dependencies)among)the)project)files))–  Default)file)to)look)for)is)makefile-or)Makefile)

27)

compiler) assembler) linker)

.c

.h

.c

project1.c*

common.h*

project2.c*

.o project1.o*

.o project2.o*

executable*Bydefiningdependencies,youcan

avoidunnecessarilyrebuildingcertainfiles.Forexample,intheexampleontheright,project2.cdoesnotneedtobere-compiledifchangeshavebeen

madetoproject1.c.

AUTOMATING THE PROCESS: MAKEFILES (2/3)

•  Makeanalyzesthe<mestampofatarget’slastmodifica<onandcomparesittothatofthetarget’sdependenciestodecidewhethertoexecutethecommand(s)definedforthattarget’srule.

target: dependencies # rule <tab> command1 # shell command <tab> command2 # shell command . .

MakefileTemplate

•  A“target”isalabel/iden-fierforarule•  OIenthetargetiseitherthenameofa

fileoraconven-onalrule(e.g.“install”)•  Dependenciesarefilesthatthetarget

dependon•  Commandsmustbeprecededbyatab

executable: project1.o proect2.o gcc –o executable project1.o project2.o

project1.o: project1.c common.h

gcc –c project1.c # generates project1.o project2.o: roject2.c common.h

gcc –c project2.c # generates project2.o

ExampleMakefile(seepreviousslide)

•  ThereareoIenmul-plerulesdefinedperMakefile

•  Byjusttyping“make”,thefirstruleinthefilewillbeexecuted

AUTOMATING THE PROCESS: MAKEFILES (3/3)

make

•  Generallybuildstheen-reproject.

makeinstall

•  GenerallyinstallsthesoIware.

maketest

•  Generallyrunsunittests.

makeclean

•  Deletesintermediatebuildfilestostartthebuildprocessfromscratch.

“makeinstall”generallyfailswith“permissiondenied”errorsifyoudonothaveadministra-veprivilegesorhavenotconfiguredthebuildto

installintoalocaldirectory.

TolearnmoreaboutMakefiles,checkoutthefollowingtutorial:

h9ps://swcarpentry.github.io/make-novice/

•  No-cethatMakeissmartenoughtonotrebuildtheprogramifnofileshavebeenmodifiedsinceourlastbuild.

•  Makeisalsosmartenoughtoonlyre-compileproject2.cwhenithasbeenchangedbutproject1.chasnot.

AUTOMATING THE PROCESS: CONFIGURE SCRIPTS (1/2)

•  AconfigurescriptisanexecutablefileresponsibleforbuildingaMakefileforaproject.

•  Determiningthedependenciesonagivensystemisdifficulttopredictandsubjecttoconstantchange–wri-ngaMakefilebyhandforeachsystem(orevenasubsetofrepresenta-vesystems)wouldbeanenormouschallengeandanadministra-vehassle.

•  Instead,aconfigurescriptcanbeusedtoscanasysteminsearchofalltheneededdependencies(includingversionsofsoIware,loca-onsofexternallibraries),andbuildaMakefilethatisspecifictothatsystem.

•  Configurescriptsareindispensibleforlargeprojectsespeciallywherethenumberofdependenciesislargeanddifficulttomanage/track.

•  Alterna-vestotheconfigurescriptexist(cmakebeingthemostcommon).

./configuremakemaketestmakeinstall

•  BuildingprojectsonLinuxat-mesthissimple.•  Runonlyifyouhaveadministra-verightsonsystem.

./configure-–prefix=/my/local/dirmakemaketestmakeinstall

•  --prefixop-onneededifinstallinginhomedirectoryonthecluster.

AUTOMATING THE PROCESS: CONFIGURE SCRIPTS (2/2)

•  Manyconfigurescriptssupportanumberofdifferentop<onsforconfiguringyourbuild.

./configure--help

•  Showcommandlineop-ons.

MAKE AND CONFIGURE MACROS

CC•  Ccompilercommand(e.g.gcc)

•  Thereareanumberof“macros”(thinkofasvariables)thathavestandardmeaningsinMakeandconfigurescripts.Thesemacroscangenerallybeexportedasenvironmentvariablestocustomizeyourbuild.

CFLAGS•  Ccompilerflags(e.g.–Wall–O3)

CPP•  Cpreprocessorcommand(e.g.gcc)

CXX•  C++compilercommand(e.g.g++)

CXXFLAGS•  C++compilerflags(e.g.–Wall–O3)

LDFLAGS•  Linkerflags(e.g.–L/path/to/lib)

LIBS•  Librarynames(e.g.–lcurl)

FC•  Fortrancompilercommand(e.g.gfortran)

FFLAGS•  Fortrancompilerflags(e.g.–O3)

MPICC•  MPICcompilerwrappercommand(e.g.mpicc)

COMPILED VS. INTERPRETED LANGUAGES

Whataboutinterpretedlanguages?

CompiledLanguage• Fasterexecu-on-me• Slowerdevelopment-me• Lessportable• C,C++,Fortran

InterpretedLanguage• Slowerexecu-on-me• Fasterdevelopment-me• Moreportable• Python,Matlab,R,Ruby,Julia

ThetradeoffslistedtotheleIarenotuniversallytruebutin

generalapply.

Manypopularmodules/packages(e.g.NumPy,SciPy)loadedfrominterpretedlanguagesarecompiledsharedobjectfilesandoffercomparableperformancetopurecompiledlanguages.