early days of message-passing computing: transputers ... · – ipsc1 had osf mach os on each node...
TRANSCRIPT
![Page 1: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/1.jpg)
Early days of message-passing computing: transputers, occam
and all that
Tony Hey Chief Data Scientist
STFC Rutherford Appleton Laboratory Harwell Campus, UK
![Page 2: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/2.jpg)
• In1981Iwasonsabba,calatCaltech–asatheore,calpar,clephysicist–andGeoffreyFoxandIwenttoacolloquiumbyCarverMead…
• CarverdemonstratedthattherewerenoengineeringobstaclestochipsgeDngsmallerandfasterforthenext20years
• IwentbacktotheUKandbuiltmessage-passingmachinesusingtheInmosTransputer
• GeoffreyFoxcollaboratedwithChuckSeitzinbuildingahypercubemessage-passingmachinethatwasusableforscien,ficapplica,ons
TheBeginnings
![Page 3: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/3.jpg)
• Designedandbuiltintheearly1980’sbyGeoffreyFoxandChuckSeitzandtheirteamsinPhysicsandCS
• Processorsatnodesofhypercube;messagepassingbetweennodes
• Experimentedwithparallelizingawholesetofscien,ficapplica,ons
• Developed‘CrystallineOS’–CrOS–whichwasreallyalibraryofcommunica,onrou,nes
• Demonstratedadvantagesofvirtualaddresses,virtualcommunica,onchannelsandkernel-likesupportateachnode
TheCaltechCosmicCube
![Page 4: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/4.jpg)
• Exploiteddataparallelismofregularproblemsby‘domaindecomposi,on’
• Forhighefficiency,needforlightweightkernelsonnodesthatallowedforlowlatencymessagestartup,mes
• Laidthefounda,onsforparallelprogrammingmethodologyandparallelperformanceanalysisthatares,llrelevanttoday
• Irregularproblemsweremoredifficult…
Lessonslearnt
![Page 5: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/5.jpg)
• TheIntelPersonalSupercomputer– iPSC1hadOSFMachOSoneachnodewhichhadveryhighlatencyforini,a,ngcommunica,ons
– iPSC2releasedsoonaZerwardswithNX‘DistributedProcess’environmentbasedonCaltech’s‘Reac,veKernel’OS
• TheTransputerSupernodemachine– BasedontheInmosT800transputerthatcombinedCPU,FPU,memoryandcommunica,onchannelsonchip
– Na,veprogramminglanguagewas‘occam’,arealiza,onofasimplifiedversionofHoare’sCSP
– EU‘Supernode’project:MachinesmanufacturedbyTelMatandParsys
TheParallelCompu7ngLandscape(1)
![Page 6: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/6.jpg)
• Manyothervendorsofparallelmessage-passingmachines:– nCUBE– MeikoCS-1andCS-2– Suprenum– Parsytec– IBMSPseries– …
Ø Eachvendorhadproprietarymessagepassingsystem
TheParallelCompu7ngLandscape(2)
![Page 7: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/7.jpg)
• ThePARMACSmacrosfromtheArgonneteam– RustyLusketal.‘Mark1’
• Thep4parallelprogrammingsystem– RustyLusketal.‘Mark2’
• TheParallelVirtualMachinePVM– VaidySunderam,AlGeistandothers– Supportedmessagepassingacrossheterogeneousdistributedsystems
• ThePARMACSmessagepassinglibraries– DevelopedbyRolfHempelandothersintheEU‘PPPE’project
PortableMessagePassingInterfaces?
![Page 8: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/8.jpg)
• In1991GeoffreyFoxandKenKennedystartedacommunityprocesstowardsadataparallelFortranstandard– ThisbecametheHighPerformanceFortraneffortandtypifiedthe‘heroic’compilerschoolofparallelprogramming
• However,whatwasclearlyneededwasalowerlevelstandardforportabilityofmessagepassingprogramsacrossdifferentparallelcomputers– TheUSwereusingp4andExpress– TheEUwereusingPARMACSinthePPPEandRAPSprojects– PVMwaswidelyusedforprogrammingnetworksofworksta,onsbutnotop,mizedformorecloselycoupledparallelmachines
TheOriginsofMPI(1)
![Page 9: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/9.jpg)
• WorkshoponStandardsforMessagePassinginaDistributedMemoryEnvironment– Williamsburg,Virginia,April1992– OrganizedbyJackDongarraandDavidWalker– SponsoredbyCRPCandKenKennedyurgedac,on
• Insummerof1992,IcontactedJackDongarraaboutstar,ngsuchastandardiza,onac,vity– DidnotwantUSandEuropetodiverge– Co-wroteafirstdraZofanMPIstandardwithJackDongarra,RolfHempelandDavidWalkerinOctober1992,nowknownasMPI-0
TheOriginsofMPI(2)
![Page 10: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/10.jpg)
OrganizedBOFsessionatSuperCompu,ng92inMinneapolis
• MPI-0documentservedasacatalyst• MarcSnirofIBMemailedmetosay‘hewashappytohavebeenplagiarized’
• IhavenoideawhyweleZtheobviouscollec,vecommunica,onsrou,nesoutofMPI-0
• RustyLuskandBillGroppfromArgonnevolunteeredtoproduceanopensourceimplementa,onoftheevolvingMPIstandard
• AndtheEUPPPEprojectpaidforthebeer…
TheOriginsofMPI(3)
![Page 11: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/11.jpg)
• FollowedproceduresofHPFForum– Setambi,ousgoalofagreeingastandardwithinoneyear
• Metevery6weeksinDallasairporthotel– IsentIanGlendinningfrommygroupinSouthamptonfundedbyEUPPPEproject
• InmyopinionMPI-1succeededbecause:– Argonneproducedanopensourceimplementa,on– Excep,onaltechnicalleadershipfrompeoplelikeMarcSnirfromIBMandJimCowniefromMeiko
– Itwasneededandhadthesupportofthecommunity
TheMPIProcess
![Page 12: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/12.jpg)
• AdventofMPImeantthatitwaspossibletoassemblesuiteofMessage-Passingbenchmarksforperformanceanalysisofmachinesandapplica,ons
• EUGenesisprojectdefined3levelsofbenchmarks– Low-level,KernelsandasetofCompactApplica,onsimplementedwithPARMACSlibraries
• Interna,onalParkbenchGroup– CombinedGenesismethodologywithLinearAlgebraandNASParallelBenchmarksimplementedwithMPI-1
Ø Butthemarke,ngcommunitypreferredJack’sTop500Benchmark…
Parkbench:PortableDMMessage-PassingKernelsandBenchmarks
![Page 13: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/13.jpg)
• ManythankstoJackDongarra,RolfHempelandDavidWalker
• Auseful‘aidememoire’wasthear,clebyDongarra,Fagg,HempelandWalkerintheEncyclopediaofElectronicsandElectricalEngineering(Wiley)
Acknowledgements
![Page 14: Early days of message-passing computing: transputers ... · – iPSC1 had OSF Mach OS on each node which had very high latency for ini,ang communicaons – iPSC2 released soon aerwards](https://reader036.vdocument.in/reader036/viewer/2022071517/613b3a72f8f21c0c8268e1fc/html5/thumbnails/14.jpg)
JackDongarra,RolfHempel,TonyHey,DavidWalker‘ADraZStandardforMessagePassingonDistributedMemoryComputers’ProceedingsoftheFiZhECMWFWorkshopontheUseofParallelProcessorsinMeteorology:“ParallelSupercompu,nginAtmosphericScience”Editors:Geerd-RHoffmannandTuomoKaurannePublishedbyWorldScien,fic,1993
MPI-0Reference