Designing Cyberinfrastructure (Jan , 2007)Paul Avery 1 Paul Avery University of Florida Cyberinfrastructure Workshop National Science Foundation Jan , 2007 Building Large-Scale Cyberinfrastructure Through Collaboration: Opportunities and Challenges Designing Cyberinfrastructure (Jan , 2007)Paul Avery 2 Open Science Grid: July 20, 2005 Consortium of many organizations (multiple disciplines) Production grid cyberinfrastructure 75+ sites, 24,000+ CPUs: US, Korea, Brazil, Taiwan Designing Cyberinfrastructure (Jan , 2007)Paul Avery 3 The Open Science Grid Consortium Open Science Grid U.S. grid projects LHC experiments Laboratory centers Education communities Science projects & communities Technologists (Network, HPC, ) Computer Science University facilities Multi-disciplinary facilities Regional and campus grids Designing Cyberinfrastructure (Jan , 2007)Paul Avery 4 Motivation: Data Intensive Science 21 st century scientific discovery Computationally & data intensive Theory + experiment + simulation Internationally distributed resources and collaborations Dominant factor: data growth (1 petabyte = 1000 terabytes) 2000~0.5 petabyte 2007~10 petabytes 2013~100 petabytes 2020~1000 petabytes Powerful cyberinfrastructure needed ComputationMassive, distributed CPU Data storage & accessDistributed hi-speed storage Data movementInternational optical networks Data sharingGlobal collaborations (100s 1000s) SoftwareManaging all of the above How to collect, manage, access and interpret this quantity of data? Designing Cyberinfrastructure (Jan , 2007)Paul Avery 5 Open Science Grid Basics Who Comp. scientists, IT specialists, physicists, biologists, etc. What Common, shared computing and storage resources High-speed production and research networks Meeting place for research groups and facility organizations Vision Maintain and operate a premier distributed computing facility Provide education and training opportunities in its use Expand reach & capacity to meet needs of stakeholders Dynamically integrate new resources and applications Members and partners Members:HPC facilities, campus, laboratory & regional grids Partners:Interoperation with TeraGrid, EGEE, NorduGrid, etc. Designing Cyberinfrastructure (Jan , 2007)Paul Avery 6 Principal Science Drivers High energy and nuclear physics 100s of petabytes (LHC)2007 Several petabytes2005 LIGO (gravity wave search) several petabytes2002 Digital astronomy 10s of petabytes2009 10s of terabytes2001 Other sciences coming forward Bioinformatics (10s of petabytes) Nanoscience Environmental Data growth Community growth Designing Cyberinfrastructure (Jan , 2007)Paul Avery 7 OSG Virtual Organizations ATLAS HEP/LHCHEP experiment at CERN CDF HEPHEP experiment at FermiLab CMS HEP/LHCHEP experiment at CERN DES Digital astronomyDark Energy Survey DOSAR Regional gridRegional grid in Southwest US DZero HEPHEP experiment at FermiLab DOSAR Regional gridRegional grid in Southwest FermiLab Lab gridHEP laboratory grid fMRI Functional MRI GADU BioBioinformatics effort at Argonne Geant4 SoftwareSimulation project GLOW Campus gridCampus grid U of Wisconsin, Madison GRASE Regional gridRegional grid in Upstate NY Designing Cyberinfrastructure (Jan , 2007)Paul Avery 8 OSG Virtual Organizations (2) GridChem ChemistryQuantum chemistry grid GROW Campus gridCampus grid at U of Iowa I2U2 EOTE/O consortium LIGO Gravity wavesGravitational wave experiment Mariachi Cosmic raysUltra-high energy cosmic rays nanoHUB NanotechNanotechnology grid at Purdue NWICG Regional gridNorthwest Indiana regional grid OSG (Misc)Generic VO OSGEDU EOTOSG education/outreach SDSS Digital astronomySloan Digital Sky Survey (Astro) STAR Nuclear physicsNuclear physics experiment at Brookhaven UFGrid Campus gridCampus grid at U of Florida Designing Cyberinfrastructure (Jan , 2007)Paul Avery 9 Search for Origin of Mass New fundamental forces Supersymmetry Other new particles 2007 ? TOTEM LHCb ALICE 27 km Tunnel in Switzerland & France CMS ATLAS Defining the Scale of OSG: Experiments at Large Hadron Collider CERN Designing Cyberinfrastructure (Jan , 2007)Paul Avery 10 CMS Experiment OSG and LHC Global Grid Online System CERN Computer Center FermiLab Korea Russia UK Maryland MB/s >10 Gb/s Gb/s Gb/s Tier 0 Tier 1 Tier 3 Tier 2 Physics caches PCs Iowa UCSDCaltech U Florida 5000 physicists, 60 countries 10s of Petabytes/yr by 2009 CERN / Outside = 10-20% FIU Tier 4 OSG Designing Cyberinfrastructure (Jan , 2007)Paul Avery 11 Crucial Ingredients in Building OSG Science Push: ATLAS, CMS, LIGO, SDSS 1999: Foresaw overwhelming need for distributed cyberinfrastructure Early funding: Trillium consortium PPDG:$12M (DOE)(1999 2006) GriPhyN:$12M (NSF)(2000 2006) iVDGL:$14M (NSF)(2001 2007) Supplements + new projects : UltraLight, CHEPREO, DISUN ($17M) Social networks: ~150 people with many overlaps Universities, labs, SDSC, foreign partners Coordination: pooling resources, developing broad goals Common middleware: Virtual Data Toolkit (VDT) Multiple Grid deployments/testbeds using VDT Unified entity when collaborating internationally Historically, a strong driver for funding agency collaboration Designing Cyberinfrastructure (Jan , 2007)Paul Avery 12 Standing on the Shoulders of Giants Previous NSF and DOE investments NMI, Globus, Condor, MonALISA, DOEGrids Certificate Authority + KX509 infrastructure Open source software components Linux (thank you, Mr. Redhat) Apache, GNU tools, myProxy, mySQL, OpenSSH, Perl, Python, Squid, Tcl, UberFTP, Wiki, etc. Technologies created by members & partners DRM, Gratia, GUMS, Netlogger, Pacman, Pegasus, PRIMA, pyGlobus, ROCKS, SRM, VDS, VOMS, Integration through the Virtual Data Toolkit (next slide) Our unique contribution Designing Cyberinfrastructure (Jan , 2007)Paul Avery 13 Integration via Virtual Data Toolkit Computer Science Research Virtual Data Toolkit Member & partner projects Grid deployments Grid & networking testbeds Science, ENG, Education Communities Globus, Condor, NMI, TeraGrid, OSG EGEE, WLCG, Asia, South America QuarkNet, CHEPREO, Digital Divide Deployment, Feedback Tech Transfer Techniques & software Requirements Prototyping & experiments Other linkages Work force CS researchers Industry U.S.Grids International Outreach Designing Cyberinfrastructure (Jan , 2007)Paul Avery 14 Communications: International Science Grid This Week SGTW iSGTW ~2 years Diverse audience >1000 subscribers Designing Cyberinfrastructure (Jan , 2007)Paul Avery 15 Project Challenges Technical constraints Commercial tools fall far short, require (too much) invention Integration of advanced CI, e.g. networks (slides) Financial constraints (slide) Fragmented & short term funding injections (recent $30M/5 years) Fragmentation of individual efforts Distributed coordination and management Tighter organization within member projects compared to OSG Coordination of schedules & milestones Many phone/video meetings, travel Knowledge dispersed, few people have broad overview Designing Cyberinfrastructure (Jan , 2007)Paul Avery 16 Collaboration with Internet2 Designing Cyberinfrastructure (Jan , 2007)Paul Avery 17 Optical, multi-wavelength community owned or leased dark fiber (10 GbE) networks for R&E Spawning state-wide and regional networks (FLR, SURA, LONI, ) Collaboration with National Lambda Rail Designing Cyberinfrastructure (Jan , 2007)Paul Avery 18 LHCNet: Transatlantic Link to CERN NSF/IRNC, DOE/ESnet (2006/2007) DEN ELP ALB ATL Metropolitan Area Rings Aus. Europe SDG AsiaPac SEA Major DOE Office of Science Sites High-speed cross connects with Internet2/Abilene New ESnet hubs ESnet hubs SNV Europe Japan Science Data Network core, Gbps circuit based transport Lab supplied Major international Production IP ESnet core, 10 Gbps enterprise IP traffic Japan Aus. Metropolitan Area Rings ESnet Science Data Network (2nd Core Gbps, National Lambda Rail) ESnet IP Core (10 Gbps) 10Gb/s 30Gb/s 2 x 10Gb/s NYC CHI LHCNet Data Network (4 x 10 Gbps to the US) LHCNet Data Network DC GEANT2 SURFNet IN2P3 NSF/IRNC circuit; GVA-AMS connection via Surfnet or Geant2 FNAL BNL CERN Designing Cyberinfrastructure (Jan , 2007)Paul Avery 19 Funding & Milestones: 1999 GriPhyN, $12M PPDG, $9.5M UltraLight, $2M CHEPREO, $4M Grid Communications Grid Summer Schools 2004, 2005, 2006 Grid3 start OSG start VDT 1.0 First US-LHC Grid Testbeds Digital Divide Workshops 04, 05, 06 LIGO Grid LHC start iVDGL, $14M DISUN, $10M OSG, $30M NSF, DOE VDT 1.3 Grid, networking projects Large experiments Education, outreach, training Designing Cyberinfrastructure (Jan , 2007)Paul Avery 20 Challenges from Diversity and Growth Management of an increasingly diverse enterprise Sci/Eng projects, organizations, disciplines as distinct cultures Accommodating new member communities (expectations?) Interoperation with other major grids TeraGrid International partners (EGEE, NorduGrid, etc.) New models of interoperation and service, e.g. caBIG Education, outreach and training Training for researchers, students but also project PIs, program officers Operating a rapidly growing cyberinfrastructure Management of and access to rapidly increasing data stores (slide) Monitoring, accounting, achieving high utilization (slide) Scalability of support model (slide) Designing Cyberinfrastructure (Jan , 2007)Paul Avery 21 Rapid Cyberinfrastructure Growth: LHC CERN Tier-1 Tier : ~140,000 PCs 3 GHz P4 ~ kSI2000 Meeting LHC service challenges & milestones Participating in worldwide simulation productions Designing Cyberinfrastructure (Jan , 2007)Paul Avery 22 Jobs Snapshot: 90 Days 5000 simultaneous jobs from multiple VOs Oct NovDec Jan Designing Cyberinfrastructure (Jan , 2007)Paul Avery 23 OSG Operations Distributed model Scalability! VOs, sites, providers Rigorous problem tracking & routing Security Provisioning Monitoring Reporting Partners with EGEE operations Designing Cyberinfrastructure (Jan , 2007)Paul Avery 24 Extra Slides Designing Cyberinfrastructure (Jan , 2007)Paul Avery 25 Cyberinfrastructure and Grids Grid: Geographically distributed computing resources configured for coordinated use Fabric: Physical resources & networks provide raw capability Ownership:Resources controlled by owners and shared w/ others Middleware: Software ties it all together: tools, services, etc. Enhancing collaboration via transparent resource sharing US-CMS Virtual Organization Designing Cyberinfrastructure (Jan , 2007)Paul Avery 26 My Background and This Talk Professor of Physics: High Energy Physics CLEO experiment at Cornell (DOE) CMS experiment at CERN (DOE) Director of two NSF/ITR Grid projects GriPhyN ( ), iVDGL ( ) Co-P.I. of several others Today: Experiences in Open Science Grid Building and sustaining national-scale grid cyberinfrastructure Serving multiple science research communities Designing Cyberinfrastructure (Jan , 2007)Paul Avery 27 Sources (CVS) Patching NMI (NSF supported) Build & Test Condor pool (~100 computers, > 20 Op. Systems) Build Test Package Build RPMs Pacman Cache Binaries Test Users Contributors Globus, Condor, myProxy, Common Middleware: Virtual Data Toolkit VDT: Package, test, deploy, support, upgrade, troubleshoot Designing Cyberinfrastructure (Jan , 2007)Paul Avery 28 ATLAS CMS LHC Global Collaborations 2000 3000 physicists per experiment USA is 2031% of total Designing Cyberinfrastructure (Jan , 2007)Paul Avery 29 Long Term Trends in Network Traffic Volumes: X/10Yrs 2005 SLAC Traffic ~400 Mbps Growth in steps (ESNet Limit): ~ 10X/4 years W. Johnston L. Cottrell Progress in Steps 2 x 10 Gbit/s TERABYTES Per Month ESnet Accepted Traffic 1990 2005 Exponential Growth: Avg. +82%/yr for the Last 15 Years 700 Designing Cyberinfrastructure (Jan , 2007)Paul Avery 30 Evolving Science Requirements for Networks (DOE High Performance Network Workshop) Science Areas 2005 End2End Throughput 5 years End2End Throughput 5-10 Years End2End Throughput Remarks High Energy Physics 0.5 Gb/s100 Gb/s 1000 Gb/s High bulk throughput Climate (Data & Computation) 0.5 Gb/s Gb/s N x 1000 Gb/s High bulk throughput SNS NanoScience Not yet started 1 Gb/s1000 Gb/s + QoS for Control Channel Remote control and time critical throughput Fusion Energy0.066 Gb/s (500 MB/s burst) 0.2 Gb/s (500MB/ 20 sec. burst) N x 1000 Gb/s Time critical throughput Astrophysics0.013 Gb/s (1 TB/week) N*N multicast 1000 Gb/s Computational steering and collaborations Genomics Data & Computation Gb/s (1 TB/day) 100s of users1000 Gb/s + QoS for Control Channel High throughput and steering See/ Designing Cyberinfrastructure (Jan , 2007)Paul Avery 31 OSG Organization Designing Cyberinfrastructure (Jan , 2007)Paul Avery 32 e-Lab and i-Lab Projects Subsumed under I2U2 program CHEPREO: Center for High Energy Physics Research and Educational Outreach Florida International University Physics Learning Center CMS Research Cyberinfrastructure WHREN network (S. America)Additional initiatives CyberBridges Global CyberBridges Networking initiatives Etc. Designing Cyberinfrastructure (Jan , 2007)Paul Avery 34 Digital Divide Effort Background ICFA/SCIC (Standing Committee on Inter- regional Connectivity) Themes Global collaborations, Grids and addressing the Digital Divide Focus on poorly connected regions Brazil (2004), Korea (2005), Poland (2006) Designing Cyberinfrastructure (Jan , 2007)Paul Avery 35 Grid Summer Schools Summer 2004, 2005, 2006 1 South Padre Island, Texas Lectures plus hands-on exercises to ~40 students Students of differing backgrounds (physics + CS), minorities Reaching a wider audience Lectures, exercises, video, on web More tutorials, 3-4/year Students, postdocs, scientists Agency specific tutorials