pacific research platform version 2 (prpv2) workshop · cenic hooks up campus gateways in a...

Pacific Research Platform Version 2 (PRPv2) Workshop

Summary of Workshop Presentations Richard Moore and Tom DeFanti

Workshop held February 21, 2017

Hosted by Calit2’s Qualcomm Institute (Calit2/QI) at

University of California San Diego (UCSD)

This workshop was supported by NSF award ACI-1541349.

Additional support is from the

University of California Office of the President (UCOP), the Corporation for Education Network Initiatives in California (CENIC), and UC San Diego Calit2

SummaryofPRPv2WorkshopPresentations Page2

TableofContentsPreface.........................................................................................................................................................3

Agenda.........................................................................................................................................................3

WorkshopContextandObjectives...............................................................................................................4

SessionNotes...............................................................................................................................................5

WelcomeandPRPTwoYearsIn–LarrySmarr(UCSD)............................................................................5

Session1:FastandSecureDataTransferPanel– Moderator:TomDeFanti(UCSD)...............................6

1.1 DataPlacementwithHTCondor–PhilPapadopoulos(UCSD/SDSC)........................................6

1.2 IPv6andCENIC/ESnet/I2Goals–JohnHess(CENIC)andTomHutton(UCSD/SDSC)..............7

1.3 FastDataTransfer:SC16results,etc.–AzherMughal(Caltech).............................................7

1.4 TrustedPlatformModules–JohnGraham(UCSD/Calit2/QI)..................................................8

1.5 Session1PanelQ&A................................................................................................................8

Session2:ScienceEngagementFuturePanel–Moderator:FrankWürthwein(UCSD)...........................9

2.1 PRPWeatherDataTransfer–ScottSellars(UCSD/SIO)...........................................................9

2.2 PRPEngagementFoci–CamilleCrittenden(UCB/CITRIS)&TomDeFanti(UCSD/Calit2/QI)..9

2.3 PRPfromtheUCSCPointofView–ShawfengDong(UCSC)..................................................10

2.4 PRPfromtheUCMercedPointofView–JeffWeekley(UCM)..............................................10

2.5 Session2PanelQ&A..............................................................................................................11

Session3:NetworkArchitectureforGlobalizationofPRPv2Panel–Moderator:CeesdeLaat(UVa)..12

3.1 CooperativeWorkGroupsinPRPv2–LeonGommans(AirFrance/KLM)..............................12

3.2 ScienceDMZsatKISTI–JeonghoonMoon(KISTI)..................................................................12

3.3 FastDataTransfer:Brazil–MichaelStanton(RNP,Brazil).....................................................13

3.4 ObservationsonExpandingthePRP–HarveyNewman(Caltech).........................................13

3.5 Session3PanelQ&A..............................................................................................................13

Session4:SDN/SDXinPRPv2Panel–Moderator:TomHutton(UCSD/SDSC).......................................14

4.1 SDNinPRPv2forsecureflows–PhilPapadopoulos(UCSD/SDSC)........................................14

4.2 100GbServicesforData-IntensiveScienceEnabledbySDN/SDX–JoeMambretti(NorthwesternUniv)..........................................................................................................................14

4.3 SDN/SDXinCENIC/PacificWave–JohnHess(CENIC)............................................................14

4.4 Session4PanelQ&A..............................................................................................................15


PrefaceThisSummaryofWorkshopPresentationsprovidesabriefoverviewoftheworkshop,includingnotesfromthetalksandpanelquestion-and-answersessions.Itisnotintendedtoreproduceorsummarizetheinformationinthespeakers’presentations,whichareavailableatpacificresearchplatform.org.Theworkshopvideo-recordingisalsoavailablethereaswell.

RecommendationsandfindingsfromtheworkshopwillbeincorporatedintotheplanningfortheremainderofthePRPproject,andmaybeaddedtothisreportatalatertime.

Agenda10:00 WelcomeandPRPTwoYearsIn–LarrySmarr(UCSD)Session1: FastandSecureDataTransferPanel–Moderator:TomDeFanti(UCSD)10:30 1.1 DataPlacementwithHTCondor–PhilPapadopoulos(UCSD/SDSC)10:42 1.2 IPv6andCENIC/ESnet/I2Goals–JohnHess(CENIC)andTomHutton(UCSD/SDSC)10:54 1.3 FastDataTransfer:SC16results,etc.–AzherMughal(Caltech)11:06 1.4 TrustedPlatformModules–JohnGraham(UCSD/Calit2/QI)11:18 1.5 Q&A11:45Lunch,withoptionalwalktoseeLibraryCAVEkioskSession2: ScienceEngagementFuturePanel–Moderator:FrankWürthwein(UCSD)1:45 2.1 PRPWeatherDataTransfer–ScottSellars(UCSD/SIO)1:55 2.2 PRPEngagementFocii–CamilleCrittenden(UCB)andTomDeFanti(UCSD/Calit2/QI)2:05 2.3 PRPfromtheUCSCPointofView–ShawfengDong(UCSC)2:15 2.4 PRPfromtheUCMercedPointofView–JeffWeekley(UCM)2:25 2.5 Q&ASession3:NetworkArchitectureforGlobalizationofPRPv2Panel–Moderator:CeesdeLaat(UVa)3:00 3.1 CooperativeWorkGroupsinPRPv2–LeonGommans(AirFrance/KLM)3:12 3.2 DMZsatKISTI–JeonghoonMoon(KISTI)3:24 3.3 FastDataTransfer:Brazil–MichaelStanton(RNP)3:36 3.4 ObservationsonExpandingthePRP–HarveyNewman(Caltech)3:48 3.5 Q&A4:00BreakSession4:SDN/SDXinPRPv2Panel–Moderator:TomHutton(UCSD)4:30 4.1 SDNinPRPv2forsecureflows–PhilPapadopoulos(UCSD/SDSC)4:45 4.2 100GbServicesforData-IntensiveScienceEnabledbySDN/SDX–JoeMambretti

(NorthwesternU)5:00 4.3 SDN/SDXinCENIC/PacificWave–JohnHess(UCSD/Calit2/QI)5:15 4.4 Q&A5:30 Reception


WorkshopContextandObjectivesThePacificResearchPlatform(PRP)isanambitiousprojectdrivenbythehigh-speednetworkingneedsofcollaborative,big-datascience.Manyresearchdisciplinesareincreasinglymulti-investigatorandmulti-institutionalandneedevermorerapidaccesstotheirultra-largeheterogeneousandwidelydistributeddatasets.Inresponsetothischallenge,theDepartmentofEnergy’sESnetdevelopedtheScienceDMZmodel,anetworksystemoptimizedforhigh-performancescientificapplicationsratherthanforgeneral-purposeorenterprisecomputing.ThePRPextendsthecampusScienceDMZmodel,whichiswidelyfundedonindividualcampusesbyNSF’sCC-NIEandCC-NIIprograms,toaregionalmodelfordata-intensivenetworking.Itenablesresearcherstoquicklyandeasilymovedatabetweencollaboratorlabs,supercomputercenters,anddatarepositories,creatingabig-datafreewaythatallowsthedatatotraversemultiple,heterogeneousnetworkswithoutperformancedegradation.ThePRP’sdatasharingarchitecture,withend-to-end10–100Gb/sconnections,alsoenablesregion-widevirtualco-locationofdatawithcomputing.

ThemainfocusofthePRPprojectistobuildaresearcher-definedanddata-focusednetworkwhoserequirementsaredrivenbydirectengagementswithsophisticated,cyberinfrastructure-knowledgeableScienceTeamschosenfromthefieldsofparticlephysics,astronomy,biomedicalsciences,earthsciences,andscalabledatavisualization.Itisapartnershipofmorethan20institutions,includingfourNationalScienceFoundation,DepartmentofEnergy,andNASAsupercomputercenters.

APRPworkshopwasheldinOctober2015,shortlyaftertheNSFawardwasinitiated,tobringtogetherrepresentativesfromallPRPpartnerinstitutions,participatingScienceTeammembers,technicalstaff,andnetworkarchitectsandimplementerstodiscussdeploymentofthePRPandaddressinteroperabilityofScienceDMZsataregionallevel(http://prp.ucsd.edu/presentations/2015-prp-workshop-report).Theprimaryobjectivesofthatworkshopwere:

• ToengagethePRPteamandcollaboratorsandfacilitatein-personinteractions.• Gainacommonunderstandingofthescienceengagementprocess.• Exploreandcapturedatastorage,computing,andnetworkingrequirementsacrossfivescientific

domainsidentifiedasinitialapplicationsforthePRP.• Identifycommonscience-driventechnicalrequirementsforthePRP.• ExchangetechnicalideasforthePRP’stechnicalimplementation.

Thetechnicaldesignandbuild-outofthePRPprojectisbeingconductedintwophases–PRPVersion1inyears1-2andPRPVersion2inyears3-5–withtheaimofachievingthefollowinggoals:

• PRPv1:Createascalablenetworkdesignforoptimizingdatatransfer.• PRPv2:EvolvetoIPv6withCooperatingResearchGroups.• PRPv2:CreateOpenFlowasafirewallandexploreothertrustandsecurityfeatures.


Considerableprogresshasbeenmadeduringthefirst17monthsoftheprojectindeployingPRPv1andsuccessfullyapplyingthatsystemtoanumberofscientificapplications.ThecurrentworkshopisfocusedonthetransitionforthePRPprojectfromVersion1toVersion2.ThePRPv1capabilitieswillnotbereplacedbyPRPv2,butratherPRPv2isanexplorationofmeanstoimprovePRPv1andpossiblyaddresstheincreasingchallengesofsecurityfromthecampuses'pointofview.ThePRPv2goalsarenotsetinstone.Thisworkshop’sgoalwasmeanttoofferanopendiscussionwithalphausers,long-termnationalandinternationalpartnersinnetworkinganddatatransfer,andtheimplementersofthePRPv1andPRPv2.Thisworkshop'spurposeistodebateanddiscussthegoalsforPRPv2.

ThePRPv2workshopwasheldatUCSD/Calit2onFebruary22,2017,inconjunctionwiththeON*VECTORInternationalPhotonicsworkshopFebruary23-24.ThePRPv2workshopwasattendedby48people,including12internationalattendeesrepresentingfivecountries(Brazil,CzechRepublic,Korea,NetherlandsandJapan).

SessionNotesThefollowingsectionisbasedoncontemporaneousnotesfromthetalksandquestion-and-answersessions.Thesenotesarenotintendedtobecomprehensive,nortoreproduceorsummarizethespeakers’presentationmaterials,whichareavailableassourcematerialatpacificresearchplatform.org.Theworkshopvideo-recordingisalsoavailablethereaswell.Thesenoteshavenotbeenreviewedbytheparticipants/speakers,andassuchtheymayinadvertentlymisrepresentthespeakers’statementsorintents;pleasecontacttheparticipantsdirectlyifyouhaveanyquestions.

WelcomeandPRPTwoYearsIn–LarrySmarr(UCSD)

ThefoundationforthePRPprojectstartedwithESnetdevelopingtheScienceDMZandDataTransferNode(DTN)concepts.

PRP’sDTNboxes,the10-100Gb/sFlashI/ONetworkAppliances(FIONAs),areacriticalenablerfortheproject.Recentlytheteamhasdevelopedalessexpensiveversion,FIONette,for~1Gbpssites.

Theprojectmonitorsdatatransferperformanceacrosstheentirethesystem4timesperday.JohnGraham,PRPchiefengineer,hasplayedacriticalroleinthedevelopment/deploymentofFIONAsandthemonitoringcapabilities.

ThePRPprojectwouldnothavebeenpossiblewithoutthelong-termsubstantialinvestmentsinCENIC.JohnHesshasplayedacriticalrolerepresentingCENICwithinthePRPproject.CENIChooksupcampusgatewaysinareliablefashion.PRPgoesfromcampusnetworkgatewaytotheresearcheroncampus.TherehashistoricallybeenagapinthisconnectivitythatthePRPistryingtofill–thecampusCIOisresponsibleforthecampusnetwork,butoftenthedepartmenthasresponsibilityfortheresearcher’sconnectivitytothatnetwork.

TherehavebeengreatstridesalreadyinPRPv1toestablish,use,andmonitorthehigh-speedend-to-endconnectivitybetweenpartnersites.(Seethecolormatricesofend-pointconnectivityperformancefromJanuary2016toDecember2016inthepresentation,aswellasreal-timeMaDDashmonitoringresultsat


http://prp-maddash.calit2.optiputer.net/maddash-webui/).Morecampuses/end-pointshavebeenaddedsincetheearlydemonstrationsoftheproject,andthereisimprovedbandwidthperformancebetweenthosesites.Theseadvancesareatributetothecollective,sustainedactivities–mostlyonavolunteerbasisinregardstothisNSFgrant-bythenetworkingpeopleatthesesites.

Therearealreadyanumberofscienceapplicationsuccessesduringthefirst1.5yearsoftheproject.Existingteamsincludethehigh-energyphysicsgroupatCaltech(HarveyNewman),ShawfengDongfromUCSCconnectingtheircampusHyadesclustertoNERSC/LBNLandgettingdatafromtheDarkEnergySpectroscopicInstrument(locatedatKittPeak~800GB/night),thevirtualrealityvisualizationcapabilityatUCMerced(JeffWeekley),andthePEERearthquakeprojectledoutofUCBerkeley.Someofthesewillbediscussedinthisworkshop.

Andtherearenewscienceusecasesthatwerenotpartoftheoriginalproposal,includingconnectingthemicroscopeoffScrippsPier(JulesJaffeUCSD/SIO),regionaldownscalingofclimatemodels(DanCayan/MikeDettingeratUCSD/SIO),expansionofHPWREN(FrankVernonUCSD),andScottSellarsandMartyRalphfromtheCenterforWesternWeatherandWaterExtremes(UCSD/SIO)usingthesystem(seelatertalk).ThereisapossibleexpansiontoincludetheOceanObservatoryInitiativein2017,connectingsensorsontheseaflooroffthestateofWashington.

Theworktodatehasalsospurrednewopportunitiesthatcouldn’thavepreviouslybeendone,withPRPprovidinghigh-speedaccesstoemergingtechnologies.Forexample,therearenewcapabilitiesbasedonnon-VonNeumannarchitectures–TrueNorthbyIBM,neuralprocessorsbyKnuEdge,field-programmablegatearraytechnologies,orARMprocessorslikeQualcomm’sSnapdragon–thatPRPcanfacilitateaccessto.Aproposalhasbeendevelopedtosupportmachinelearningcapabilities,usingacloudofGPU-heavyFIONAboxesinaCondor-managedcloud.Andtheconceptisexpandingtointernationalcapabilities,ledbysomeoftheworkshop’sinternationalparticipants.

Thistalkhasreviewedthepastwork–i.e.PRPV1.ThisworkshopfocusesonPRPV2–lookingaheadtoSDN/SDX,IPVv6,andothertechnologies.

WeshouldthankthePRPpartners–inMissouriterms,theteamhasbeen‘raisingabarn,’basedlargelyonpartners’volunteerlaborcontributingtothesegoals.

Inresponsetoaquestionaboutsustainedfunding,LarryindicatedthathehastalkedwithprogramofficersatNSFandthefundingprospectsareuncertainbothatanagencylevelandattheOfficeofAdvancedCyberInfrastructure(ACI).ThereissomereorganizationwithintheOfficeofACI–NSFislookingforanewOfficedirector,andthereareprioritydebateswithintheofficeacrosssupercomputers,dataefforts,softwareandnetworking.HehopestherewillbeongoingfundingopportunitiesthatwouldleveragethelargeinvestmentsNSFhasalreadymade,andnotesthatInternet2hasissuedwhitepaperscitingnetworking’sroleinmajorinfrastructureprojectsforthecountry.

Session1:FastandSecureDataTransferPanel– Moderator:TomDeFanti(UCSD)

1.1 DataPlacementwithHTCondor–PhilPapadopoulos(UCSD/SDSC)

ThetalkdiscussesaprojectthatisrelatedtoPRPwithDr.MironLivnyatUWisconsin-MadisonandotherpartnersatBeihangUniversityandtheComputerNetworkInformationCenter(CNIC)inChina.


PRPisoftenfocusedonrawperformance(bandwidth).Thisprojectextendsthemetricstoreliabilityandaccesstodata.Moresecurityissuesareinvolvedwhentouchingdisks,andisnetworkmeasurementperformanceagoodproxyfordisk-to-diskperformance?Thisprojectmeasuresend-to-endanddisk-to-diskperformanceamongasetofinternationalendpointsusingdifferentdata-movementprotocolsandIPv4andIPv6.

Therearealotofknownnetworktraversals,butit’strickytoknowwhattheactualpathsare.

A‘testmanifest’issubmittedasaroutinechronjobtoaCondorpool,withHTCondorhandlingerrors/recovery/reporting/iteration.AlimitedCondorpoolisbuiltacrosssites,withnodesneedingtotrusteachother‘enough.’Serversaresetupandtorndowninuserspaceforexperiments.

Seeperformanceslidesinthepresentation.Manyoftheend-point/protocolresultsshowsubstantialvariabilityinperformance,whichisnotnecessarilyunderstoodatthispoint.ThetestingdidindicatethatiRODSperformanceis~10%oftherawnetworkperformance.

Commonly-usedcomponentsinclude:HTCondor,Graphite,CarbonandWhisperdatabase(open-sourcedfromOrbitz),Python>2.6.x,Git;seeslidesforwebsites.AllcomponentshaveV4andV6implementations.

1.2 IPv6andCENIC/ESnet/I2Goals–JohnHess(CENIC)andTomHutton(UCSD/SDSC)

PRPv1isnotuniqueintermsofnetworkingarchitecture.

ThereisageneralconcernoncampusesaboutScienceDMZsbypassingcampusfirewalls.Wedon’twanttoinadvertentlycreateinsecurepaths.AlsotheprojectandcampuseswanttoensurethatthePRPnetworkisappropriatelyusedforresearchpurposesonly.

What’senvisionedforPRPv2?PRPv2willreflectadifferentstrategy,usingBorderGatewayProtocol(BGP)communitiesfortaggingclassesofDMZnetworks.Theplanistohaveapilotimplementationacrosssixsites(UCSD,SDSC,Caltech,NCSA,UChicago…andoneNorthernCaliforniasite).PilotBGPpeeringwillbenativeIPv6only.AstretchgoalistoincorporateSDN/SDXtypesignalingforpathsorsuper-channels.

Takethe~20sitesinPRPv1andtake~6PRPv2pilotsites.UserouteserverslocatedatPacificWavesites.Tagroutesforthispurpose.Dynamicsignaling.Dataplanewilltraverseexistingpaths.

ProposingIPv6asamechanismtodealwithestablishingspecificroutesconnectinggroups–generallythereisIPv6addressspaceavailable.AndoneoftheobjectiveshereistoaccelerateadoptionofIPv6.

ESNethasfourpublicDTNsthatareavailableforread-onlytransfers.PRPmaywanttoconsiderasimilarapproach.

Tstatisawaytoinstrumentdatatransfers(thinkofitasNetflow++).Theywouldliketogetsomemajorcenterstoincorporatetstat,butitwon’tbeeveryone.Andwouldliketobeabletogeneratereportsonthevariousconnectionsviatstat.

TomDeFanticlarifiesthatPRPv1isnotgoingaway,butratherwillpersistthroughthenextphaseoftheproject,withPRPv2beingaparalleltestbed.

1.3 FastDataTransfer:SC16results,etc.–AzherMughal(Caltech)

TalkfocusesondemonstrationsconductedatSC’16andplansforSC’17.Seepresentationmaterialsforgoalsofdemonstrations.

Theteamhadafew1Tpbslinksacrosstheshowfloor(Caltech,StarLight,SCinet),more100Gbslinks.


SC16demosacrossCENIC:PRPbasedontopoftheCENICnetworkbackbone.Oneoftheobjectivesisto‘energize’thescienceteamssothatresearcherscanseewhatcanbeachieved.

1Tbpsbooth-to-boothtransfers:

• LookedatvariousNVMedrivesforhigh-speedtransfers.Builtalow-costDTNserverthatcoulddrive~100Gbps.Maximumthroughputat14drives(7drives/processor).

NeedprocessorwithmorePCIelanesinordertogethigherspeeds.IntelSkyLake?

Haveachievedtransmissionacross4MellanoxVPINICs–closeto400Gbps.

SC17goals

§ EastWestintegrationwithothercontrollersalongwithstate,recovery,provisioning,monitoring§ DemonstratingSENSEprojectforDTNautotuning(SENSE=SDNforend-to-endnetworked

scienceatexascale)§ NVMeoverFabricsacrosstheWAN§ DTNdesignusing200GNICs(Mellanox/Chelsio)

1.4 TrustedPlatformModules–JohnGraham(UCSD/Calit2/QI)

TrustedPlatformModules(TPMs)offerfacilitiesforthesecuregenerationofcryptographickeys,andlimitationoftheiruse,inadditiontoarandomnumbergenerator.TPMsarelow-costdevices,allowingsystemstointeractsecurely.

HavesuccessfullyimplementedUC-JupyteronSDSC’sCometHPCsystemusingTPMs.JupyterHubauthenticatesauserwithCILogonandspawnskernelsonComet.TrustedPlatformModules(TPM)ontheJupyterHubFIONAssecurethekeysfromtheCILogonmemberorganization.Thesekeysareusedtoconnectjupyter.calit2.optiputer.nettocomet.sdsc.edu.

GlobusandXSEDEhaveCILogonauthentication,soapproachcanextendtomanysystems/users.

HowdoyouscaleaccesstoJupyternotebookscampus-wide?Berkeleyhasbeendoingit.DoingapilotnowwithProfColemanatUCSD.

Developingnext-generationGPUJupyterbox.Dual8-coreCPUs,8GPUs/box,6x480GBSSDs,2NVMebays,~$13K.

DistributedTrustedComputing.UsingKubernetes,aspinoffofGoogleinfrastructuretools;seeslidesforlistoffeatures.Takecontainer,signcontainerwithTPM,thennoonecantamperwithanythingincontainer.

KubeSpawner(jupyterhub-kubernetes-spawner).Canspawnsingle-usernotebookserversonaKubernetescluster–multipleentitiesresidentonsamehardware.

1.5 Session1PanelQ&A

TherewasaquestionforJohnGrahamabouttheSSDsandcoolingissuesinthenext-genGPUbox.Johnindicatedthereare6SSDdrivesand2NVMebays.ThereareeightGPUsandfourfans.Johnnotesthattheunithasbeenordered,butnotrunyet–itwillbeinterestingtoseehowcoolandhowloudtheboxis.Itwillbeinterestingtoevaluatethecooling,buttheyhavehadsimilarboxesandexpectittowork.AzherMughalnotesthattheyhavehadtolookathigh-speedfansfortheirboxes.

Inresponsetoaquestiononliquid-cooledoptions,JohnGrahamrepliesthathehasseensomeliquid-cooledGPUs,butlow-profileunitsarerequiredtofitinthebox.


TherewasaquestionforAzherMughalaboutwhetherspecialconfigurationswererequiredforhigh-speedtransfers.Yes,thesystemshadtobeconfiguredandtunedtoachievehigh-speeds.

TherewasaquestionforPhilPapadopouloswhythesetupforIPv6wasmorepainfulthanIPv4intheirtesting.TheprimaryissueswerelimitationsontheUCSDcampus.BecausethereisnotthatmuchexperiencewithIPv6oncampus,therearenotverymanyIPv6servicessetupyet.Forexample,ittookawhiletogetrequirednetworkinfo,routesarestatic.Thisshouldimprovewithmoreexperience.

CeesdeLaatnotesthattheIPv6servicesavailabletohimlocallyinNetherlandsareprettyrobust.Philwasjealous.

TomHuttonnotesthatbecauseIPv6isoptionalratherthanrequiredformanypeople,itkeepsgettingdeferred.Incontrast,Phil’sChinesecollaboratorsonlyhadIPv6addressspace,sotheyhadtogetuptospeed.ThetentativeplanatUCSDisprobablytosetupasub-domainwithintheDMZOptiputernetworkthat’sIPv6.

GeorgePeekaskedageneralquestion–fromaPRPperspective,what’sthebiggestheadacheforIPv6?TomHuttonandPhilPapadopoulosreplied‘Applications.’JohnHessnotedthatsysadminsmaybereluctanttosetuprulesetsforIPv6.

JoeMambrettiaskedJohnHessaboutIPv4andIPv6issuesforvariousinstitutionsinPRPv2.JohnrepliesthatamongexerciseswithNCSAorUChicago,thecollectionofrouteshavegreatresourcesbutwe’reonlyinterestedinthesubsetofroutesthatmaptoaddressesforScienceDMZandhigh-performanceresources.Mostlargeinstitutionsmayhaveroutesthatarebothv4andv6.WanttomakenetworkIPv6friendlyenoughthatresearchersdon’tneedtoworryaboutIPv4orIPv6configurations–i.e.usersinitiaterequesttomovedataandthenetworkworks.Thetechnicalendisofteneasier,whilethescienceengagementpiececanbemorechallenging.

Session2:ScienceEngagementFuturePanel–Moderator:FrankWürthwein(UCSD) FrankWürthweinposedthreequestionsforthespeakersinthissession–notingthatthespeakershavenotbeeninformedofthesequestionsinadvanceoftheirtalks!

• HowcanPRPv2addvaluetoyourwork?• IsPRPv2goingtoserveITorscienceneeds,orboth?

o AreyouconcernedthatITandscienceneedscouldbeatoddswitheachotherinPRPv2?• HowcanwemakechangeleastdisruptivetoexistingPRPv1scienceuse?

2.1 PRPWeatherDataTransfer–ScottSellars(UCSD/SIO)

Hisgroupishandlinglargeclimatologydatasets(seepresentationforlist).

Theyhadarecentworkflowtoidentifyfeaturesinwatertransfermodels–tookabout20daysfora4-stepprocesswithmuchofthatbeingnetworkdatatransfertime.

TheyhaveworkedwithPRPandotherstoexpeditethisworkflowandimprovetheirscientificproductivity.Thedownloadspeedincreased~4XusingPRPnetworkandFIONAs.TheyhavealsobroughtinKeplerworkflowsoftware,gotconnectedtoSDSC’sCometsystem,andareusingJupyter.

2.2 PRPEngagementFoci–CamilleCrittenden(UCB/CITRIS)&TomDeFanti(UCSD/Calit2/QI)

PRPwillbeparticipatinginanumberofworkshops:


• Add-ontoannualCENICconferenceMarch20,2017.• BigDataandEarthSciencesMay30-June2withScottSellars.• Inplanning:Cryo-EM,machinelearning,HPWREN.• Thereissomeadditionalparticipantsupportfundingavailableinthegrant.

Thereareanumberofadvantagesofparticipatingintheseworkshops,includingexposureandpotentialexperiencetostudentparticipants.

ThePRPalsoconvenesregularPRPnetworkingtelcons–whilesomepartnersparticipateregularly,allpartnershaveaccesstonoticesandminutesofthesemeetings.

Theprojectwouldliketodobetterataninstitutionallevel(comparedtonetworkinglevel)andincrackingthenutofscienceengagementacrosscampuses.PRPwantsmorefaculty,postdocsandstaffonboard.

Scienceengagementeffortsdobetterwhenyoucanhighlightscienceopportunities,e.g.incontextofworkshops.Engagementsneedperson-to-personcontact,andneed‘evangelists’(beyondLarrySmarr)totalkwithpeople.Theprojecttriedtorecruitsomeonetofillthisrole(e.g.apostdoc)andranintoobstacles.Evangelismremainsachallengefortheproject.

Workshopparticipantsareencouragedtoreachouttoscientistsontheircampuses.Networkingproblemsaretough–andaregettingattentionoftalentedpeople;stillneedhelponthesocialengineeringside.

Howdoestheprojectteamknowit’sbeingsuccessful–whenemailsaboutproblemsstop?That’sanimportantmeasure,buttherearebroaderissuesbeyondthatmetric.GenerallyITpeopleonacampusareinadifferentrealmthanprofessorsandresearchstaff.It’simportanttointroducefacultytotheITstaffthatcanhelpthem.

2.3 PRPfromtheUCSCPointofView–ShawfengDong(UCSC)

Heworksin‘ResearchandFacultyPartnerships(RFP),’adivisioninthecampusITorganizationthatisresponsibleforworkingwithfaculty/researchersatUCSC.Hehimselfisanastrophysicist,withexpertiseinHPCsimulationsforhisownresearch.Ithelpshiminhisroleasinterfacetofacultythathehasastrongresearchbackground.

SeepresentationforaslideonrelatedRecentAwardsatUCSC–theyincludeaCC-NIE$500KawardforScienceDMZanda$400KCC-DNIawardforCIEngineer(fundinghim).

Theirpartnersacrosscampus:ComputationalAstrophysics,Astronomy,ParticlePhysics,Genomics,Chemistry,Biology,DataScience.Dominatedbyastrophysics,butaregettingotherpeopleoncampusengaged.

2.4 PRPfromtheUCMercedPointofView–JeffWeekley(UCM)

HeistheDirectorofCyberinfrastructureandResearchComputingatUCMerced.

OneofhisobjectivesistoelevateUCMtoafirst-classparticipantinPRP.

TheyarebuildingcyberinfrastructureatUCMerced:

• BuiltaWAVE.Interestingly,intheSchoolofHumanitiesandtheArts.• RecipientofCC-*awardforScienceDMZ.IntendtosupportIPv6.• CampusisplanningtodoubleitssizeinProject2020-$1.3Bconstructionproject.Shouldbeable

tobuildnetworkingintherightway.


• Planningwireless/microwavelinksoutsidethecampus.• MaybeHPWREN2.0–Yosemite-MontereyBay.Nationalparkservice,UC,StateofCalifornia,

otheragencies.

UCMercedhasfacultythathavesignificantnetworkingneedsinanumberofpioneeringapplications.

AnotherobjectiveistobuildHumanCapacity.Theyareremovingroadblocks,repeatingworkflows,fosteringtouchstoneprojectsacrossUCcampuses,partneringwithCITRISandCalit2,anduncovering‘force-multipliers.’


ScottSellarswasaskedaboutthecomputingenvironmentforanalysishe’sdoingonhisdata.AtthispointitisprimarilyPC-levelhardwarewithtoolssuchasMatlabandPython.TheymaymovetocentralHPCresourcestospeeduptheanalysissteps.Hewasalsoaskedaboutthetotalreductionintimefromtheoriginal20-dayworkflow.Thenetworkingimprovementsreduceditto~10days.Theymayalteralgorithmsandmovetocentralcomputingtoreduceitfurther.

TomDeFantiwasaskedaboutPRPdemonstrationsplanned,inadditiontoworkshops,asameanstoengagewithsciencegroups.Tomnotesthattherehavebeensomekeydemonstrations,includingtheeffortsattheannualSupercomputingconferences.

Thepanelwasaskedaboutmorespecificideasforworkshopparticipantstobetterengagefaculty,includingeffortsthatwouldscaletolargenumbersoffaculty.CamilleCrittendenhighlightedtheconceptofforcemultipliers–e.g.researchersthatareinvolvedinresearchITeffortsalready.Speakingatconferencesisgoodforachievingscale.AnotherideaistoaskresearchersthathaveusedPRPtoacknowledgeitinpublications,justastheywouldgrantfunding.TomDeFantinotesthatoneapproachistopicksomeoneinasectorandfocusresourcesonthemto‘makethemfamous.’Thenthatperson’scolleagueswillhearwhatcanbeachievedbyleveragingsomethinglikePRP.JeffWeekleycommentsthathepitchesPRPandbigdatamoversandScienceDMZ–nomatterwhatthetopicwhenhe’stalkingwithfaculty.Also,ifhecan’tgettofaculty,hegetstogradstudents.CamilleCrittendennotedthatagoodtimetoengageiswhennewfacultymembersarejustjoiningtheuniversity.ShawfengDongcommentedthattheirengagementhasbeenhelpedbythefactthathisbossisanadjunctfacultymember,andShawfengpersonallyhasbeenpartofresearchgroups.

LaterinthediscussionHarveyNewmannotedthatpartofscienceengagementistoshowpeoplewhatothersareabletoachievecomparedtowhatthey’vebeendoing…andhencehowmuchtimethey’rewasting.

TimLanceaskedaboutengagement/evangelizationvia‘listeningversustalking.’JeffWeekleycommentedthathelistenstowhatpeopleneedanddoesn’tbuildwhatisn’tneeded.TomDeFantinotedthathetalkedwithresearchersinmachinelearninginordertocomeupwithideaforthehardwaretosupportit.

ThesessionmoderatorFrankWürthweinwasaskedabouthisprogressinestablishinghigh-speedend-to-endconnectivityforLHChigh-energyphysicsresearchersacrossUCandothercampuses.Histeamhasputboxesat~5campuses(Davis,Irvine,RiversideSanDiego,soonSantaBarbara)andtheyareallconnectedat10Gbps.Theyhavebenchmarkedtheinfrastructureandjustrecentlysubmittedaresearchpaperontheproject.UCIrvineisprobablymakingthemostscienceuseofthecapabilityrightnow.Intermsoflessonslearned,hehadthoughtitwouldbe‘cookiecutter’deployments,butfoundoutthatallthecuttersandcookieswerealittledifferent.Itwasalsohardertogetadoptionthanexpected-


thereisanotherstepbetweengettingworkingequipmentatasite,andtrainingthelocalresearchershowtooperateanduseit.

ShawfengDongandJeffWeekleywereaskedtowhatextentmulti-campusormulti-nodalcollaborations(withincampus)areimportanttoresearchontheircampuses.ShawfengDongrepliedthattheLHCcollaborationwasimportanttocampuseffort,andthePRPeffortreallymadeadifferenceinreducingsignificantbottlenecks.JeffWeekleycharacterizedmulti-campuscollaborationsasanexampleofaforcemultiplierthathehadnotedinhistalk.Hehasmadeinvestmentsthatenablecampus->edgecapabilitiesandthatwillmakeadifference.He’sseenexamplesofothercampusesthatjusthaveawfulnetworkingconnectivityandit’sarealbarriertocollaboration/research.

ThepanelwasaskedaboutusingpublicDTNsthatareclosetoPOPs(similartowhatESNethasdone).ThegeneralreplywasthatDTNsatPOPSareagoodidea,andthatitwouldbeanexcellentideafortheretobecentralizeddatarepositorieswithhighperformance.Oneparticipantnotedthatusersstillneedtobeinformedhowtodiscoverandaccessthoseassets.

ScottSellarswasaskedwhetherhisprimarychallengeisgettingthedatainthefirstplaceorwhereitgoesafterhegetsit.Scottrepliedthatit’sachallengethatthecentraldatasetsaresolarge,sinceonehastomakedecisionsinadvancewhichportionofthedatatodownload.Iftheinfrastructureexistedtoeasilypulldatadownentiredatasetsandthenprunedata,hewouldpreferthatapproach.

Session3:NetworkArchitectureforGlobalizationofPRPv2Panel–Moderator:CeesdeLaat(UVa)

3.1 CooperativeWorkGroupsinPRPv2–LeonGommans(AirFrance/KLM)

‘TrustedSharingofBigDataAssetsinCooperativeWorkingGroups.’

Therearebarriersforbusinessorganizationstosharedata–e.g.wouldothersthatyou’resharingwithgetmorevaluerelativetoyourorganization,intellectualpropertyandlegalconcerns,andprivacyprotections.

‘SecureDigitalMarketPlace’conceptallowscommunitymemberstosharedataaccordingtowell-definedmarketrules.

Usinghigh-speedbandwidthofthePRPtoenablein-memoryanalysesofshareddata,withoutstoringtheshareddata,wouldhelpalleviatesomeofthedata-sharingconcerns.Couldalsoutilize‘containers’onthatdatawithinthesharedspace.

Wantstodevelopa‘futureinternetfieldlab’(seeslides).

3.2 ScienceDMZsatKISTI–JeonghoonMoon(KISTI)

ThespeakernotesthatDMZhasaveryreal–andnon-scientific-meaninginKorea.

HaveanSDN-basedDataTransferNodeandaLayer3-basedDTN(FIONAbox).ThelatterwasusedtojointheSC’16demonstrationdescribedearlierbyAzherMughal,andthisnodewasjustaddedtothePRP‘MaDDash’matrixofend-points.

Theyhaveachieved9.2Gbps(FTP)and9.1(GridFTP)on10GbpslinefromSeoultoDaejeon.DaejontoPRPuniversitiesinCaliforniaachieves7-8GpbsbothdirectionswithGridFTP.

OneoftheirscienceusecasesisAgroMeteorologyFederation(AMF)inKorea,storingclimatedataandanalyzing/sharingresults.


Doingresearchtodevelopan“OpenScienceDMZCloud”withDTN/DMZasaService(DDaaS).Theyarealsoworkingonaprojecttodevelopa“SaaSOverCloudonDTN.”

3.3 FastDataTransfer:Brazil–MichaelStanton(RNP,Brazil)

RNPisBrazil’sResearchandEducationNetwork.TheRNP’snationalbackboneconnectscapitalsofeachstateandfederalcapital(26capitalsand13othercities);mostlinksareat10G,exceptintheAmazonregion.Maymovesomeoftheseto100Ginthenextyear+.TherearealsosharedlinkstotheUSviaASNPandAmLightEXP,EuropeandLatinAmericaviaRedClara,andterrestriallinkswithArgentina.Thereisaconcentrationofuniversities/researchcenters/populationinSaoPaolo/RiodeJaneiroregions.

RNPhasparticipatedinSCdemonstrationssince2004.InSC’16(withCaltechgroup),RNPdemonstrated>95%usageofthe100GaccesstoUS,viaANSP.

It’simportanttotalkaboutusingdata,notjustmovingit(andnotjustdemos).ABraziliane-astronomylabcollaborateswithNCSAandJHU/Sloandataviahigh-speedlinks.Whentheystarted,theygot20-30Mbpsona1Gbpsline,andittooktwoweekstotransfera5.7TBcollection.Nowtheyget~1GbpsusingGridFTP(andaDTNboxessentiallyatthePOP).AndthereisanewprojecttoenableefficientremoteaccesstoLNCC’ssupercomputer(inPetropolis).PotentialclientsarealightsynchrotronsourcesandaWeather/ClimateLab.Thiscapabilitymaysavecostofpurchasingaseparatecomputerforthesynchrotron.

Henotesthathehasseeninterestingworkhereonhigh-speedDTNsthathewillbetakingbacktoBrazil.

3.4 ObservationsonExpandingthePRP–HarveyNewman(Caltech)

Thisisthetimetoexpandhorizons,tothinkinthelarge.ThereareprogramsonthehorizonthatwillbeevenbiggerthanLHC–e.g.LSSTandSKA.

Historically,ESnettraffichasgrown~10Xover4years.MuchofESNettrafficisLHCdata.InpastyearalonewithLHCRun2,LHCdatagrowth~2.7X.

Ontheotherhand,wehavenotbeeninaperiodofhightechnologicaladvancement.We’rein4th-5thyearof~100Gbpstechnology.Nexttechnologystepcomingis~400Gb/s.Thatrateoftechnologyimprovementisnotkeepingpacewithtrafficgrowthrates.

SC16demo–upto1Tbpslocal,many100Gbpsconnections.

ThereisamajoropportunityforanewCPU/storage/networkecosystem+LargeComputingFacilities(LCFs)asfocalpoints.

SENSEeffort:“SDNforend-to-endnetworkedscienceatexascale”withpartnersESNet,Caltech,Fermilab,Argonne,MarylandandLBNL.

PRPhasbecomealocusforactivityinthisarea.


Thepanelwasaskedaboutfuturemajordrivers.HarveyNewmanprovidedexamplessuchasLSST,genomics,andtheInternetofThings.

TherewasaquestionaboutthelatencyofthepathtoKoreaintheSC16demonstration-itapproached400msecround-trip.

Formanyworkflows,datainarepositoryisshippedsomewhereforanalysis,andthenresultsarereturnedtotheresearcher.Thequestionwasposed-whatarethecomputingenvironmentsatthe


endpoints?HarveyNewmanrepliedregardingtheLHCcomputingenvironment–e.g.~20Tier1sites,~150Tier2sites(eachwithseveralpetabytesstorageandseveralthousandjobslots).Datatransfersaretypically~2TB;ifyoucouldmove10sofTBsefficiently,thatwouldsignificantlyincreaseproductivity.

TherewasaquestionabouttheBrazilianAmazonriversprogram,thatwasprimarilydeferredtothefollowingON*VECTORworkshop.MichaelStantonsaidthattheyhavecompleted~250kmthepastyear,andexpecttosetupnext400kmwithinnext4-5months.

LeonGommanswasaskedforadditionalinformationonairlinedatasharing.Leoncitedtheexamplethatwhenanaircraftlands,thereis~500GBofdataavailableperflight.Withafleetofaircraftoveralongperiodoftime,therecouldbeexabytesofdatatheoreticallyavailable.Canthisdatabesharedeffectively?

Session4:SDN/SDXinPRPv2Panel–Moderator:TomHutton(UCSD/SDSC)

4.1 SDNinPRPv2forsecureflows–PhilPapadopoulos(UCSD/SDSC)

ThetalkaddressesusinganOpenFlowSwitchasalow-impactfirewall.

InLinux,everypacketislookedat,meaningthereisoverheadoneverypacket.ButLinuxcan’tmakedecisionsbasedonother‘meta’data.

PurposehereistouseanOpenFlowswitchtomakedecisionsbasedonflows,reducingoverheadperpacket.Butloseobservabilityofflowfromcontroller.Limitedflow-tablespace.Andwhathappensifcontrollergoesoffline–networkgoesaway.

Theytestedhowmuchdelaycouldbeintroducedininitiatingrulesfortheflowwithoutimpactingnetworkperformance:0.5sec(alot!)wasalmostnoimpact,while5seccripplednetwork.SoitlooksreasonabletouseanOpenFlowswitchasafirewall.

ThereareadditionalpossibilitieswithIPv6.

4.2 100GbServicesforData-IntensiveScienceEnabledbySDN/SDX–JoeMambretti(NorthwesternUniv)

‘OpenFlowisdead.’Orchestrationiscritical.Highlyprogrammablenetworks.LotsofimportantworkcomingoutofGENI.

SDX’sareinUS,butwhereelse?SomeatSURFNet,onesooninGermany,oneotherinEurope.WillbeinTaiwan,Singapore,Korea,Tokyo.

ThereweremanydemosatSC16,mostdependentonSDNtechniques.

HelikesmdtmFTPcomparedtoGridFTP.

WantstomovetowardstheGlobalResearchPlatform–aworldwidePRP.

4.3 SDN/SDXinCENIC/PacificWave–JohnHess(CENIC)

PacificWaveisajointcollaborationCENICandPacificNorthwestGigapop(PNWGP),supportingbothcommercialandR&Epeers,servingcountriesacrossthePacificconnectingtothewesternUS.

PacificWavereceivedanNSF/ACIIRNCawardSeptember2015tosupportmore100GconnectionsandtodeploySDN/SDXonparallelinfrastructuretoenableexperimentationwhilemaintainingproductionuseofthePacificWaveexchange.AnSDXnodeisbeingestablishedinLosAngeles,andanexistingSDXnodeinSeattleisbeingleveraged;controlnodesarelocatedinSeattleandLosAngeles.


Thegroupisparticipatingintheworldwide‘AutoGOLE/NSIPilot’project.(GOLE=GLIFOpenLightpathExchanges,GLIF=GlobalLambdaIntegratedFacility.)


JohnGrahamnotedthattheGlobusfeaturesforsharingdata,citedinoneofthecharts,costmoneycomparedtothebasicGridFTPdata-movementcapabilities.JohnHessrepliedthatyes,Globushasalicensingmodelthatenablessomeadditionalfeatureslikemultiplestreams,datasharing,andthird-partyauthentication.ESnethasanetwork-widelicenseforGlobus,andGlobushasallowedthemtousethislicenseforthepilotproject.Theyhopetogetasimilararrangement…butnotclearyethowfaritcanbeextended.

PhilPapadopouloswasaskedwhethertheOpenFlowswitch-as-firewallcouldbeusedforalldata.Herepliedthatit’sthe‘niceflows’(nottheshorthousekeepingflows)thatwouldgothruthissystem.

JohnHesswasaskedhownewinstitutionscanjointheSDN/SDXeffort.Heindicatedthatifaninstitutionisinterested,besttostarttheprocessbytalkingwithhim.

pacific research platform version 2 (prpv2) workshop · cenic hooks up campus gateways in a...

Documents