nics and the nsf's high-performance computing...
TRANSCRIPT
![Page 1: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/1.jpg)
NICS and the NSF's High-Performance
Computing Program
Jim Ferguson
NICS Director of Education, Outreach & Training
8 September 2011
![Page 2: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/2.jpg)
Acknowledgement – Phil Andrews, 1955 - 2011
NICS Director, pre-opening of NICS through February 2011
Studied Physics at Cambridge, Purdue, Princeton
Worked at Pittsburgh Supercomputing Center and San Diego Supercomputer Center before moving to the National Institute of Computational Sciences.
2
![Page 3: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/3.jpg)
National Institute for Computational Sciences University of Tennessee and ORNL partnership
NICS is the 2nd NSF Track 2 center
• Building on strengths of UT and ORNL
• Systems are now in production
Series of computers culminating in a 1.2 PF system in 2011
3
![Page 4: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/4.jpg)
NICS Resources
Kraken
• legendary sea monsters of gargantuan size said to have dwelt off the coasts of Norway and Iceland
4
![Page 5: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/5.jpg)
How Big is The Kraken?
112,896 compute cores residing on 9,408 nodes.
147 terabytes of compute memory.
3.3 raw petabytes of parallel file system.
Theoretical peak performance of 1.17 petaFLOPs (FLoating point OPerations per second).
![Page 6: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/6.jpg)
6
![Page 7: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/7.jpg)
NICS Timeline
7
NSF Track-2 Award Announced Sep 2007
First Cray XT3 Resources Available May 2008
Cray XT3 Production – 40 TF Jun 2008
Cray XT4 Acceptance Jul 2008
Cray XT4 Production – 18K cores - 166 TF Aug 2008
Cray XT5 Delivery Dec 2008
Cray XT5 Production – 64K cores – 615 TF Feb 2009
100M Hours Awarded at TRAC Mar 2009
Cray XT5 Upgrade – 100K cores – ~1 PF Nov 2009
Cray XT5 Upgrade – 112K cores -- ~1.2 PF Jan 2011
![Page 8: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/8.jpg)
Other significant NSF Awards at NICS
NSF TeraGrid XD center: Remote Data Analysis & Visualization (RDAV). $10M/3-year award to UT, main computing resource to be an SGI ‘Ultraviolet’ machine. Sean Ahern, Principal Investigator
NSF Track 2D ‘Experimental’ system: Main computing resource will be a GPU-based hybrid system. Award through Georgia Tech, Principal Investigator is Jeffrey Vetter
Kraken memory upgrade
Extreme Science and Engineering Discovery Environment (XSEDE): Just announced, $18M/5-year
8
![Page 9: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/9.jpg)
Access to NSF computational resources
Done by Peer Review process at quarterly Resource Allocation meeting.
Prospective Principal Investigators apply, then are reviewed by peers, and allocations made among the NSF resources available.
Directors at each Center also have “discretionary” time that they may dole out.
“Startup” accounts for researchers to try out computing resources are available easily.
Educational allocations are usually approved easily.
9
![Page 10: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/10.jpg)
IGMCS Access
UT students registered for the IGMCS program will get an allocation on NICS resources
Details being finalized, likely at least 20,000 hours for each student
10
![Page 11: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/11.jpg)
How did we get Here? The NSF Supercomputing Timeline
Pre 1984: scramble for cycles
First Proposals: unsolicited appeals to NSF for national level resources, 1983-1984
Centers Program: 1986-1996
PACI Program: 1997-2004
TCS+DTF+ETF = TeraGrid: 2000-2011
Track 2: 2007-2010 (A – D)
Track 1: 2011
TeraGrid XD: 2011 (“XSEDE”)
11
![Page 12: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/12.jpg)
Technology Changes
Initial systems in the 1980s were all vector processors (Cray or ETA) which favored vectorizable code
Today’s systems are all “Massively Parallel” or clusters which favor scalable code.
Along the way, we’ve seen several different architectures, including Single Instruction Multiple Data, Multi-threaded, etc.
Already arrived: General Processing Graphics Processing Units and other accelerators. Offer great speedup with lower power requirements, but are currently difficult to program efficiently.
12
![Page 13: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/13.jpg)
Seymour Cray
13
![Page 14: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/14.jpg)
NSF HPC Early Years
Little “Open” capability: NMFECC for Fusion
Early 1980s: Studies evaluating need for access to supercomputers.
Program Advisory Committee at NSF, chaired by Neal Lane including 25-30 computational scientists, frequent meetings.
Unsolicited proposals from Princeton (Steve Orszag), Illinois (Larry Smarr) and UC-San Diego (Sid Karin), Oct 83 – Feb 84.
Later 1984: NSF formal solicitation, 22 proposals received.
14
![Page 15: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/15.jpg)
Early Years (continued)
Early 1985: Three Center awards announced: San Diego Supercomputer Center, National Center for Supercomputing Applications, and Jon von Neumann Center
“Advanced Prototype” award to Cornell, who later became a Center.
Late 1985: Friendly users at Centers. Official openings in early 1986.
Early 1986: Award to Pittsburgh Supercomputing Center.
15
![Page 16: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/16.jpg)
What about the Network?
NSFnet formed in 1986-1987 to connect the Centers. NSF enforced the adoption of TCP/IP standards for networking. This worked out!
NCSA Telnet provided full TCP/IP stack for PC, Macintosh, and Sun workstations.
NSFnet was eventually released by NSF and transitioned to private hands, the beginnings of today’s commodity internet (1995).
vBNS, a fast network research network connecting the Centers, debuted the same day NSFnet went commodity.
16
![Page 17: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/17.jpg)
Open Software development
NCSA Telnet: a model for “open source” tools.
Gplot, Gdoc and P3D from PSC
SDSC and NCSA Image Tools
NCSA Mosaic: the ultimate NSF Center software tool
17
![Page 18: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/18.jpg)
18
![Page 19: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/19.jpg)
What is different today?
Back then: supercomputers were like tall buildings – monolthic and designed as a unit, unique and difficult to change.
Now: supercomputers are generally made up of smaller components, and pieces can be added or subtracted at any time.
The problems of modern centers are essentially civil engineering issues: weight, power, and heat dissipation.
19
![Page 20: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/20.jpg)
NSF Reorganizes
2005: Supercomputing was moved out of CISE and into the Office of Cyberinfrastructure (OCI) reporting directly to the NSF Director.
Separate research award programs, SDCI and STCI, established.
Other Directorates at NSF have strong input into directions of OCI. Resulted in more service oriented (and shorter), prescriptive awards.
20
![Page 21: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/21.jpg)
Tracks 1 and 2: A new beginning
Four Track 2 RFPs announced, yearly awards, $30M captial equipment, operations funded separately.
One Track 1 award, $200M for “sustained PetaScale system”, production target 2011.
Very prescriptive on allowed work. Production oriented, very low operations funding.
TeraGrid connectivity important.
21
![Page 22: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/22.jpg)
Track 2 Awards
2A: Texas Advanced Computing Center, recommended 2006, awarded 2007.
• Sun system, AMD processors, custom switch.
2B: National Institute for Computational Sciences, recommended & awarded 2007.
• Cray XT5, AMD processors, SeaStar interconnect.
2C: Pittsburgh recommended 2008, never awarded due to problems with main vendor (SGI).
2D: San Diego, Georgia Tech, Indiana awarded smaller “experimental” systems.
Dear Colleague Letter awards in 2010 to NICS, TACC, NCSA, SDSC, PSC for a variety of systems & upgrades.
22
![Page 23: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/23.jpg)
![Page 24: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/24.jpg)
Track 1 Award
Awarded to the National Center for Supercomputing Applications in 2007.
• IBM PERCS system (Power Series)
• Sustained PetaFlop
• >10PF, 1 PB memory, 10PB disk, 1EB storage.
• 100 Gb/s WAN connectivity
• IBM Cancelled contract (through an Out clause) in August 2011, NCSA negotiating a new deal with new vendors.
24
![Page 25: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/25.jpg)
What’s Next?
TeraGrid XD: 2.5 years in the making
Two major proposals went in, NSF asked that they be blended (roughly 85/15)
New program, “XSEDE”, has major partners NCSA, PSC, TACC and NICS.
Researchers who use the current TeraGrid should see little or no disruption to current practice.
25
![Page 26: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/26.jpg)
September 8, 2011
XSEDE: National Computational Science and Education
Services
![Page 27: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/27.jpg)
What is XSEDE?
27
XSEDE is a comprehensive set of advanced, heterogeneous
high-end digital services, integrated into a
general-purpose infrastructure.
![Page 28: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/28.jpg)
What does that mean?
28
![Page 29: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/29.jpg)
XSEDE Vision
The eXtreme Science and Engineering Discovery Environment (XSEDE) will:
enhance the productivity of scientists and engineers by providing them with new and innovative capabilities.
Thus, XSEDE will:
facilitate scientific discovery while enabling transformational science and engineering, and innovative educational programs.
29
XSEDE will fulfill this vision by creating an advanced, capable, and robust cyberinfrastructure supported by the combined expertise of a distributed team of
leading CI (cyberinfrastructure) professionals.
![Page 30: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/30.jpg)
XSEDE Characteristics:
• XSEDE forms the foundation of a national cyberinfrastructure (CI) ecosystem
– Its comprehensive suite of advanced digital services will federate (combine) with other high-end facilities and campus-based resources
• XSEDE integrates diverse digital resources
– Its open architecture allows continued addition of new technology capabilities and services
30
![Page 31: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/31.jpg)
XSEDE is about . . . . • Increasing productivity
– leading to more science – making the difference between a feasible project and an
impractical one • Transformative impact through
– active, formal requirements gathering processes to • understand the needs of the community
– new and expanded advanced support that includes • external short-term contracting for expertise beyond
the current team • Novel and Innovative Projects: supports novel science
areas, demographic diversity, innovative technologies, science gateway development, data repositories, and campus bridging
31
![Page 32: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/32.jpg)
And… – National Training and Education and Outreach programs with
the scope and scale to: • increase diversity of topics, modes of delivery, and reach to
new communities and audiences • broaden participation among under-represented
communities • campus bridging for effective use of CI (cyberinfrastructure)
resources – integrate with campuses through expanded Champions
program and additional bridging activities • establish certificate and degree programs
– institutional incorporation of CS&E curricula; professional development certificate
– prepare undergraduates, graduates and future K-12 teachers
32
![Page 34: NICS and the NSF's High-Performance Computing Programigmcs.utk.edu/sites/...XSEDE-UT_seminar-2011-09-08.pdf · 9/8/2011 · XSEDE Characteristics: •XSEDE forms the foundation of](https://reader033.vdocument.in/reader033/viewer/2022050209/5f5c42d78acc3b308c732374/html5/thumbnails/34.jpg)
Who works at a place like NICS?
System Administrators. Large HPC systems, security systems, file systems, job scheduling, mass storage systems, networks, user account administration, etc.
User Support. Front-line answers for users, documentation, FAQ collection, web site management, user surveys, training, allocation management, diagnosing problems.
Computational Scientists. Second-line, in depth help for users. Experts in various disciplines, training, documentation, diagnosing problems.
Management. Reporting, proposal writing, hiring
Outreach & Education.
External Relations.