Acommonandsustainablebigdatainfrastructureinsupportofweatherpredictionresearchandeducationinuniversities

Unidata ModelingResearchintheCloudWorkshop,5/31/17

CarlosMaltzahn Background• CurrentResearch

• High-performanceultra-scalestorageanddatamanagement

• End-to-endPerformancemanagementandQoS

• ReproducibleEvaluationofSystems

• NetworkIntermediaries

• OtherResearch• DataManagementGames• InformationRetrieval• CooperationDynamics

2

• AdjunctProfessor,ComputerScience,UCSantaCruz

• Director,UCSCSystemsResearchLaboratory(SRL)

• Director,CenterforResearchinOpenSourceSoftware(CROSS)cross.ucsc.edu

• Director,UCSC/LANLInstituteforScalableScientificDataManagement(ISSDM)

• 1999-2004:PerformanceEngineer,Netapp

• Advising 6Ph.D.students.• Graduated 5 Ph.D.students• Idothis100%ofmytime!

Project• NSF-fundedScientificSoftwareIntegrationproject(SSI)oftheSoftwareInfrastructureforSustainedInnovation(SI2)program• Goal:sustainablecommunitySWframework

Collaborators

CarlosMaltzahn,IvoJimenez(UCSantaCruz),

JoshHacker,JohnExby,KateFossell (NCAR),

MohanRamamurthy(Unidata),

GretchenMullendore,TimothySee(UND),

BrianAncell (TexasTech),

WilliamCapehart (SDSM),

ClarkEvans(UWMilwaukee),

RobertFowell,KevinTyle (UAlbany),

StevenGreybush (PennState),

RussSchumacher(CSU).

ProblemInformedbyEarthCube Usersworkshops• Poorreproducibilityofdata-intensivescience

• Impactoneducationandresearch

• Impairedavailabilityofintermediateresults• Unnecessaryduplicationofwork,steeplearningcurves

• Communitiesofpracticearefallingbehind• Limitedabilitytoadoptnewtechnologies

4

Domain:NumericalWeatherPrediction• NWPgroupsatuniversitiesusesupercomputingtimetocreatelargeensembles• Currentpractice:

• keepensemblesinscratchspaceordownloadtolocalinfrastructure• Don’tshareensembleproducts,don’tsharetools• Rewardsforresults,notdata

5

GeneralApproach

• Establish“nuclei”:piecesoftechnologythat• Areeasilyshareable• Havetheabilitytogrow&improveovertime• Ensure“buy-in”fromresearchersandstudents

• Examples:• Wikipedia• Linuxkernel

• Infrastructurestoenablecommunity-drivenreviewandimprovement

6

BigWeatherWebNuclei

1. Largeensembledistributedover7universities:GretchenMullendore (UND),BrianAncell (TexasTech),WilliamCapehart (SDSM),ClarkEvans(UWMilwaukee),RobertFowell (SUNYAlbany),StevenGreybush (PennState),RussSchumacher(CSU).

2. Commonstorage,linking,andcatalogingmethodology:DataInvestigationandSharingEnvironment• Permanentnamingandhighavailabilityofdataandexperiments• Connectingdata,platform,tools,analysis

3. SoftwareContainertechnologiesforeasydeploymentandreproducibility• Self-contained:softwarecanbeinstantlydeployedincommonenvironments• Namingandversioning:compactreferencemechanismsforcomplexenvironments• Goodforreproducibilityandeducation

7

Nucleus1:Large,distributedensembles

• Testingthedistributedensembleframeworkandtools• Sharingof“knowledgeproducts”

• Initializationmethods• Physicsoptions• Workflowscriptsforproducing&analyzingdata• Success:BWWPis areusingtheBWWensembletodoscience

• Trackingdataauthorshipandcommunityimpact• WehaveaDOIbutaccesshastobemanaged(expenseofdataegress,seebelow)• Ensembleisevolvingovertime

• Disseminationofframework&tools• SeeNCAR’s“WRFinabox”work

Nucleus1:Large,distributedensembles

• Testingthedistributedensembleframeworkandtools• Sharingof“knowledgeproducts”

• Initializationmethods• Physicsoptions• Workflowscriptsforproducing&analyzingdata• Success:BWWPIsareusingtheBWWensembletodoscience

• Trackingdataauthorshipandcommunityimpact• WehaveaDOIbutaccesshastobemanaged(expenseofdataegress,seebelow)• Ensembleisevolvingovertime

• Disseminationofframework&tools• SeeNCAR’s“WRFinabox”work

Nucleus2:Commonstorage,linking,andcatalogingmethodology• Enablefiguresinpublications&teachingmaterialstolinktoenvironments,tools,anddatathatproducedthem• Providedinaformthatisreusable

• Easyinstallofenvironmentandtools• Creationandaccesstodataproductswithoutneedtodownloadeverything• Dataproductsbythemselveslinkbacktotheirantecedentsinareusableway.

Usecloudservicesinsteadofon-premise installations

• Convertshardtechnical,management,andfundingquestionsintojustfundingquestions• Startedwithcommercialcloud:AWS($9k/monthcredit)

• 50TBsofaronS3:$800/month• THREDDSserveronEC2:$160/month• ParticularthankstoJohnExby andKevinTyle

• Challenges:• Commercialcloud:costofdataegress.PlanningmovetoXSEDE/TACC/Wrangler• Long-termmanagementofstoragecommons(withbetter-than-scratch-spacepolicies)

• Long-termnaming:gettingaDOIistheeasypart-- long-termavailability?

11

Nucleus2:Commonstorage,linking,andcatalogingmethodology

Nucleus3:SoftwareContainers

• Seeearliertalk”CollaborativeWRF-basedresearch&education,enabledbysoftwarecontainers”byJoshHacker,JohnExby,andKateFossell• PracticalFalsifiableResearch(Popper,falsifiable.us,seeposter)

• Applyopen-sourcesoftwarecommunitypracticestoexperimentmanagement• Scripteverything,leverageworkflowsystems&DevOpstools

• SeealsoEricKlavins’AquariumProject:klavinslab.org/aquarium.html• Keepeverythingingit repositories• Usesoftwarecontainersforallsoftware

• Namingconventiontoautomaterunningandvalidatingexperiments• Conventionsforcompactcomputingenvironmentdescription• SeeposterbyIvoJimenez

12

BWWOutreach

• 2015Unidata UsersMeeting• 2015AGUTownhall

• ~50attendees• 10newbww-userssubscribers

• 2016PresentationatAMS• 2016Unidata Workshops• WRFinaboxintheclassroom

• 2016UNDclassbyTimSee(UND)• 2017UNDclassbyGretchenMullendore (UND)• Wiliam Capehart (SDSM)usingBWWensembles(2papers)

• Popper• Jimenezetal.VarSys’16,Chicago,IL• Jimenezetal.USENIX;login:,Winter’16• Guestlecturein2017UNDclassbyGretchenMullendore

13

• Websites:• bigweatherweb.org• www.ral.ucar.edu/projects/ncar-docker-wrf• falsifiable.us

• Emaillist:bww-users@unidata.ucar.edu• Slack:bigwxweb.slack.com,invitations:KevinTyle,ktyle@albany.edu• Contact:CarlosMaltzahn,carlosm@ucsc.edu