award no. 1541215 4/1/2016 nsf-arlington, va · award no. 1541215 aristotle overview & goals...
TRANSCRIPT
AwardNo.1541215 4/1/2016NSF-Arlington,VA
AwardNo.1541215
BuildingaFederatedCloudModel:AristotleCloudFederation
DavidLifka,PIDirector,CornellUniversityCenterforAdvancedComputing(CAC)InterimCIO&VicePresident,CornellInformationTechnologies
ThomasFurlani,Co-PIDirector,CenterforComputationalResearch(CCR),UniversityatBuffalo
RichWolski,Co-PIProfessor,ComputerScience,UniversityofCalifornia,SantaBarbara
Co-Founder,[email protected]
AwardNo.1541215
AristotleOverview&Goals• $5million,5-yearNSFawardtodeployafederatedDIBBs
cloudatCornell,U.Buffalo,UCSantaBarbara– 7scienceteams,over40globalcollaborators
• Requiringflexibleworkflows/analysistoolsforlarge-scaledata• Representingdiversityofanalysisrequirementsandcloudusagemodalities
– Earthandatmosphericsciences,finance,chemistry,astronomy,civilengineering,genomics,foodscience
• Projectgoals– Optimize“timetoscience”– Demonstratethevalueofsharingresourcesanddataacross
institutionalboundaries
3
AwardNo.1541215
34
5
6
1
2
7
Multi-DisciplinaryCollaborationsinaMulti-CampusCloudFederation
AwardNo.1541215
ProjectWorkBreakdownStructure
5
CloudFederation Project Management,Oversight&Reporting
CU0.10FTEUB0.01,UCSB0.01TotalFTE:0.12
DIBBsAcquisition,Installation,Configuration,Testing&Maintenance
CU 0.08,UB0.25,UCSB0.1TotalFTE:0.43
CloudFederation PortalCU 0.33,UB1.0,UCSB0.66
TotalFTE:1.99
PortalDevelopment&MaintenanceCU 0.28,UB0.25,UCSB0.20
TotalFTE:0.73
Allocations&AccountingIntegrationCU 0.02,UB0.1,UCSB0.1
TotalFTE:0.22
OpenXDMoDIntegrationCU 0.02,UB0.65,UCSB0.0
TotalFTE:0.67
Eucalyptus&QBETSIntegrationCU 0.01,UB0.0,UCSB0.36
TotalFTE:0.37
ResearchTeamSupportCU 0.30,UB0.75,UCSB0.31
TotalFTE:1.36
AwardNo.1541215
ProjectGovernance
6
DirectorNSFProgramManager
ExecutiveSteeringCommittee
ScienceTeamAdvisoryCommittee(STAC)
ExternalAdvisoryCommittee(EAC)
UserCommunity
AwardNo.1541215
ExternalAdvisoryCommitteeIntroductions
7
Name Affiliation Project Email
Amy Walton NSF Aristotle [email protected]
Adam Brazier CAC Aristotle [email protected]
Susan Mehringer CAC Aristotle [email protected]
Paul Redfern CAC Aristotle [email protected]
Resa Reynolds CAC Aristotle [email protected]
Jamie Kinney AWS SciCo [email protected]
Dmitrii Zagorodnov HPE Eucalyptus [email protected]
Ben Rosen Dell Big Data & Cloud [email protected]
Craig Stewart IU Jetstream [email protected]
John Towns NCSA XSEDE [email protected]
Rick Wagner SDSC Comet [email protected]
Ian Foster UC Globus [email protected]
Steve Johnson WCM NIH CTSC [email protected]
AwardNo.1541215
ProjectOfficeUpdate(Q1/Q2)• CompletedCooperativeAgreement&PEP(12/18/15)
– EstablishedExternalAdvisoryCommittee• CompletedUBandUCSBsubcontracts• Establishedbiweeklyteamcalls• Establishedmonthly/quarterlyreportingprocess• Discussionsaroundenablingtechnologies
– OpenXDMoDdatacollectionforAristotlefederation@SC15– CycleComputingleverage@SC15– SupercloudwithCornellComputerScienceandJetstream– QBETSintegrationintoOpenXDMoDandGlobusGenomics– Keyportalcomponentsandcapabilities– AWSSciCopartnership
• Agreementonallocationsandaccountingprinciples
8
AwardNo.1541215
InfrastructureUpdate(Q1/Q2)• CornellRedCloudinproduction for5years
– RedCloudEucalyptusupgradedto4.2– FederationcapabilitiestestedbetweenIthacaandWCMNYC– CurrentAristotleportalsiterunninginRedCloud– January2016firstinstallmentofDIBBinfrastructureordered/installationcomplete
• UBLakeEffectcloudinstalled/operational10/16/15– LakeEffectEucalyptusupgradedto4.2– IntegratingintoexistingUBcoreservices– LeveragingEucaAvailabilityZones– January2016firstinstallmentofDIBBsinfrastructureordered/installationnearlycomplete
• UCSBEngineeringComputing Infrastructurecloudinproduction for2years– UCSBEucalyptuscloudupgradedto4.2– WorkingonInCommonsetup– FirstinstallmentofDIBBsinfrastructureordered/arrivingnow
9
AwardNo.1541215
PortalUpdate(Q1/Q2)• Goal:InformedBursting
– AcrosstheAristotlefederation– NSFClouds– AWS
• Definedportalcomponentsanddeveloperresponsibilities• CompletedInCommonintegrationproofofconcept
– SupportfromHPEHelionEucalyptusteamforInCommonaccesstoEucaPortalindevelopment• CompletedsetupofAristotleRTsystematCornell
– [email protected]– Establishedticketroutingroles
• CompletedAllocations&Accountingdatabaseschemaanddatabase– BasedonstreamlinedversionofCornellCACAllocations&Accountingsystem
• WilldrawbasicuserdocumentationfromRedCloud• Utilizationandallocationbalancereporting
– AbilitytoshowDIBBsusagevs.localinstitutionalusage• IntegrationofXDMoDandQBETs• GlobusOnlineupatCornell;UBandUCSBinprogress
10
AwardNo.1541215
ResearchTeamsUpdate(Q1/Q2)• 7initialscienceteams
– UseCase1:ACloud-BasedFrameworkforVisualization&AnalysisofBigGeospatialData(UB)– UseCase2:GlobalMarketEfficiencyImpact(UB)– UseCase3:HighFidelityModelingandAnalyticsforImprovedUnderstandingofClimate-
RelevantAerosolProperties(CU)– UseCase4:TransientDetectioninRadioAstronomySearchData(CU)– UseCase5:WaterResourceManagementUsingOpenMORDM(CU)– UseCase6:MappingTranscriptomeDatatoMetabolicModelsofGutMicrobiota(CU)– UseCase7:Multi-SourcedDataAnalyticstoImproveFoodProduction(UCSB)
• Scienceteammeetingsstartedfromday1– Understandcurrentworkflowsandprocesses– Documentsoftwareandhardwarerequirements– Developcloudadoptionstrategy– StartcreatingVMs(e.g.,geospatialanalytics,financeanalytics,Windowsinstancefor
MATLABVMsalreadycompleted)– Demoproofofconceptstoscienceteams
11
AwardNo.1541215
UseCase1:ACloud-BasedFrameworkforVisualizationandAnalysisofBigGeospatialData(UB- Chandola)
• What?– Integrate,visualizeandanalyzeheterogeneousgeospatialdatafrom
aroundtheworld• WhyAristotle?
– Offloadvariablecomputationaldemands fromlocalinfrastructure– Createanexpertservicehubforwidelydistributeddata– Scalableandelasticcomputing
12
AwardNo.1541215
UseCase2:GlobalMarketEfficiencyImpact(UB- Tin,Roesch,Wolfe)• What?
– Measuremarketefficiencieswithhighvolumesofhistoricalandcurrentintradaymarketdata
• WhyAristotle?– Bursttointegratenewdata– TensofTBsmovedtoavailableprocessingresources– Sharedataandpipelinesaroundtheworldforcollaboration
andreproducibility
13
AwardNo.1541215
UseCase3:High-FidelityModelingandAnalyticsforImprovedUnderstandingofClimate-RelevantAerosolProperties(CU- Pryor)
• What?– Modelthebehaviorofatmosphericaerosolparticlestopredictand
ameliorateclimateandhumanhealthimpacts• WhyAristotle?
– Bursttoprocessnewdata– Shareahigh-valueprocesseddatasetofgeneralinterest– Buildan extensibletoolkitforresearchers,transferrabletoother
groups
14
AwardNo.1541215
UseCase4:TransientDetectioninRadioAstronomySearchData(CU- Cordes)• What?
– Low-latency/on-demandsearchfortime-varying,includingone-time,signalsinradioastronomydata
– BuildonexistingCornell/CACcollaboration*• WhyAristotle?
– Burston-demandtoanalyzenewdata– High-speedtransportofverylargedatasets intoandaroundthe
federation
*"ArepeatingFastRadioBurst,”Nature531,202–205(10March2016)
15
AwardNo.1541215
UseCase5:WaterResourceManagementUsingOpenMORDM(CU- Reed)• What?
– Simulatecomplexwaterresourcemanagementoutcomeson-demandtoachievesubstantialbeneficialenvironmentalimpactonmunicipalitieslocatedthroughouttheUS
• WhyAristotle?– Startcoarselyparallelcomputationson-demand– Verylargecomputations, idealfortransfertoAWSorotherclouds– Makeavailabletomanymunicipalities
16
AwardNo.1541215
UseCase6:MappingTranscriptomeDatatoMetabolicModelsofGutMicrobiota(CU- Douglas)
• What?– Usehostandgut-microbialexpressiondatatoinferwhichmetabolic
reactionsoccurineachorganismasaresultofdietaryinputs• WhyAristotle?
– Heterogeneous instancetypesandsizeswithhighavailabilitytoallowforunpredictablecomputationaldemands
– AllowsgenerationofreproduciblecomputationalbiologypipelinesintheformofVMsorVMconfigurations
17
AwardNo.1541215
UseCase7:Multi-SourcedDataAnalyticstoImproveFoodProduction(UCSB/SedgwickReserve- McCurdy)
• What?– Interdisciplinaryanalysisofdatafromawidevarietyofsources
(includingthegeneralpublic)tostudyandimproveecologicaloutcomes
• WhyAristotle?– Scalableandportableinfrastructure– Accesstomultipledatasources,manyofwhicharealreadyin
publicandprivateclouds
18
AwardNo.1541215
OutreachActivities• CASCPresentation
LifkapresentedinfoonAristotleatthefallCoalitionforAcademicScientificComputation(CASC)meetinginArlington.Severalinstitutionsexpressedimmediateinterestinjoiningthefederation.Lifkaexplained thatiscertainlyagoalandwewillreportbacktoCASCwhenweareready.
• MediaOutlets/SocialMediaGeneratedmediacoverageacross23globalmediaoutletsincludingmajorIToutlets (CIO,CampusTechnology,GovernmentTechnologyNews,NetworkWorld)aswellasHPCverticals(ACMTechNews,HPCwire,InsideBIGDATA, ScientificComputingWorld).Astrongsocialmediapresence(Twitter,LinkedIn,etc.)wasgeneratedincollaborationwithNSF,AWS,andotherprojectpartners.
• CloudLaunchBackgrounderCornellproducedaCloudLaunchbackgrounderthatdescribesitsfeaturesandbenefits:https://www.cac.cornell.edu/technologies/CloudLaunch.pdf.
• SC15ConferenceTheAristotlefederationwasfeaturedatCornell’sandUB’sSC15exhibitsandnews andbackgroundersweredistributed toattendees.
19
AwardNo.1541215
OutreachActivities(cont.)• NationalMediaCoverage
ACMTechNews- CornellLeadsNewNSFFederatedCloudProjectCampusTechnology-CornelltoLeadNSF-FundedCloudFederationforBigDataAnalysisCloudwards-Aristotle:AcademicFocusedCloudFundedCIO- Universityresearchersget$5Mgranttobuild'AristotleCloud'CloudStrategyMagazine-CornellLeadsNewNSFFederatedCloudProjectDataCenterTalk-CornelltoHeadthe$5MFederalCloudComputingProgramGlobalWirelessResearch-Universityresearchersget$5Mgranttobuild'AristotleCloud'GovernmentTechnologyNews-WisdomoftheClouds:AristotleCloudFederationHPCwire-CornellLeadsNewNSFFederatedCloudProjectInsideBIGDATA-CornelltoLeadAristotleCloudFederationforResearchNetworkWorld-Universityresearchersget$5Mgranttobuild'AristotleCloud'NextGenerationCommunications-CornellLeadsNewNSFAristotleCloudFederationProjectScientificComputing-Cornelltolead$5MNSFFederatedCloudProjectScientificComputingWorld-Willthecloudchangescientificcomputing?WebHostIndustryReview-NationalScienceFoundationSponsors$5MillionCloudProject
• InternationalMediaCoverageCIOL(India)-CornellUniversitytodevelopfederatedcloudComputerWorld(Australia)-Universityresearchersget$5Mgranttobuild'AristotleCloud'PrimeurMagazine(UK)-CornellleadsnewNationalScienceFoundationfederatedcloudprojectTechWorld(Australia)-Universityresearchersget$5Mgranttobuild'AristotleCloud'
• LocalMediaCoverageCornellChronicle-Aristotle:AfederatedcloudforacademicresearchCornellDailySun-CornelltoLead$5MFederalCloudComputingProgramIthacaJournal-Cornelltoheadefforttostreamlinedataflows
20
AwardNo.1541215
SharingandWorkingwiththeCommunity• We’redevelopingandidentifyingcloud-relatedtools
– QBETShttp://www.cs.ucsb.edu/~nurmi/nurmi_qbets.pdf
– OpenXDMoDwithcloudsupport http://xdmod.sourceforge.net/
– CloudLaunch(potentialworkwithCycleComputingandAWS)https://www.cac.cornell.edu/technologies/CloudLaunch.pdf
– Supercloud(testingonJetstream)http://fireless.cs.cornell.edu/publications/supercloud.pdf
• Wewanttocreatea”Cloudy”communityfocusgroup– Identifyandsharecloudtools,services,andsolutions– Leveragenewcomputingandscalingparadigms(e.g.,containerization)– Helpeachotherandthegreaterresearchcommunityachievefastertimeto
science
21
AwardNo.1541215