Developments in Supercomputing

Ron PerrottQueen’s UniversityUnited Kingdom

2

H. Meuer, H. Simon, E. Strohmaier, & J. Dongarra

- Listing of the 500 most powerfulComputers in the World

- Yardstick: Rmax from LINPACK MPPAx=b, dense problem

- Updated twice a yearSC‘xy in the States in NovemberMeeting in Germany in June

- All data available from www.top500.org

Size

Rat

e

TPP performance

Top500 List of Supercomputers

Performance Development

0.1

1

10

100

1000

10000

100000

1000000

10000000

100000000

1 Gflop/s

1 Tflop/s

100 Mflop/s

100 Gflop/s

100 Tflop/s

10 Gflop/s

10 Tflop/s

1 Pflop/s

100 Pflop/s

10 Pflop/s

59.7 GFlop/s

400 MFlop/s

1.17 TFlop/s

8.2 PFlop/s

41 TFlop/s

59  PFlop/s

SUM

N=1

N=500

6-8 years

Laptop (12 Gflop/s)

1993 1995 1997 1999 2001 2003 2005 2007 2009 2011

iPad2 & iPhone 4s (1.02 Gflop/s)

November 2011: The TOP10Rank Site Computer Country Cores Rmax

[Pflops]% of Peak

Power[MW]

MFlops/Watt

1 RIKEN Advanced Inst for Comp Sci

K computer Fujitsu SPARC64 VIIIfx + custom Japan 705,024 10.5 93 12.7 826

2 Nat. SuperComputer Center in Tianjin

Tianhe-1A, NUDT Intel + Nvidia GPU + custom China 186,368 2.57 55 4.04 636

3 DOE / OS Oak Ridge Nat Lab

Jaguar, Cray AMD + custom USA 224,162 1.76 75 7.0 251

4 Nat. Supercomputer Center in Shenzhen

Nebulea, DawningIntel + Nvidia GPU + IB China 120,640 1.27 43 2.58 493

5 GSIC Center, Tokyo Institute of Technology

Tusbame 2.0, HP Intel + Nvidia GPU + IB Japan 73,278 1.19 52 1.40 850

6 DOE / NNSA LANL & SNL

Cielo, Cray AMD + custom USA 142,272 1.11 81 3.98 279

7 NASA Ames Research Center/NAS

Plelades SGI Altix ICE 8200EX/8400EX + IB USA 111,104 1.09 83 4.10 265

8DOE / OS

Lawrence Berkeley NatLab

Hopper, CrayAMD + custom USA 153,408 1.054 82 2.91 362

9Commissariat a

l'Energie Atomique (CEA)

Tera-10, Bull Intel + IB France 138,368 1.050 84 4.59 229

10 DOE / NNSALos Alamos Nat Lab

Roadrunner, IBM AMD + Cell GPU + IB USA 122,400 1.04 76 2.35 446

November 2011: The TOP10Rank Site Computer Country Cores Rmax

[Pflops]% of Peak

Power[MW]

MFlops/Watt

1 RIKEN Advanced Inst for Comp Sci

K computer Fujitsu SPARC64 VIIIfx + custom Japan 705,024 10.5 93 12.7 830

2 Nat. SuperComputer Center in Tianjin

Tianhe-1A, NUDT Intel + Nvidia GPU + custom China 186,368 2.57 55 4.04 636

3 DOE / OS Oak Ridge Nat Lab

Jaguar, Cray AMD + custom USA 224,162 1.76 75 7.0 251

4 Nat. Supercomputer Center in Shenzhen

Nebulea, DawningIntel + Nvidia GPU + IB China 120,640 1.27 43 2.58 493

5 GSIC Center, Tokyo Institute of Technology

Tusbame 2.0, HP Intel + Nvidia GPU + IB Japan 73,278 1.19 52 1.40 865

6 DOE / NNSA LANL & SNL

Cielo, Cray AMD + custom USA 142,272 1.11 81 3.98 279

7 NASA Ames Research Center/NAS

Plelades SGI Altix ICE 8200EX/8400EX + IB USA 111,104 1.09 83 4.10 265

8DOE / OS

Lawrence Berkeley NatLab

Hopper, CrayAMD + custom USA 153,408 1.054 82 2.91 362

9Commissariat a

l'Energie Atomique (CEA)

Tera-10, Bull Intel + IB France 138,368 1.050 84 4.59 229

10 DOE / NNSALos Alamos Nat Lab

Roadrunner, IBM AMD + Cell GPU + IB USA 122,400 1.04 76 2.35 446

500 IT Service IBM Cluster, Intel + GigE USA 7,236 .051 53

Geographical regions

Count Share % Rmax Rpeak Cores

North America 272 54.40% 32923947 48374869 4659645

Eastern Asia 109 21.80% 25868736 38046465 2520930

Western Europe 49 9.80% 8020850 10532996 1173728

Northern Europe 36 7.20% 3652751 5071283 428832

Eastern Europe 11 2.20% 1482188 2519402 126856

Southern Europe 7 1.40% 665279 1047276 60904

Western Asia 6 1.20% 530526.6 808867.6 115540

Australia and New Zealand 4 0.80% 353753.5 479797.9 35424

South America 2 0.40% 269730 330444.8 37184

South-central Asia 2 0.40% 187910 242995.2 18128

Southern Africa 1 0.20% 61330 74257.9 6336

South-eastern Asia 1 0.20% 52633 98995 9304

Sums 500 100% 74069633.68 107627649.54 9192811 6

South America HPC

Rank Site System Cores Rmax (TFlop/s)

Rpeak (TFlop/s)

Power (Kw)

49INPE (National Institute for Space Research)

Tup - Cray XT6 12-core 2.1 GHz 30720 205.1 258

Brazil Cray Inc.

290NACAD/COPPE/UFRJ

Galileu - Sun Bladex6048, Xeon X5560 2.8 Ghz, Infiniband QDR 6464 64.6 72.4 430

Brazil Sun Microsystems

7

Japanese K Computer

8New Linpack run with 705,024 cores at 10.51 Pflop/s (88,128 CPUs)

China• First Chinese Supercomputer to

use a Chinese ProcessorSunway BlueLight MPPShenWei SW1600 processor, 16 core, 65 nm, fabbed in China125 Gflop/s peakIn the Top20 with 139,364 cores & 1.07 Pflop/s Peak

• Coming soon, Loongson (Godson) processor

8-core, 65nm Loongson 3B processor runs at 1.05 GHz, with a peak performance of 128 Gflop/s

9

Commodity plus Accelerator

10

Intel Xeon8 cores3 GHz

8*4 ops/cycle96 Gflop/s (DP)

Nvidia C2070 “Fermi”448 “Cuda cores”

1.15 GHz448 ops/cycle

515 Gflop/s (DP)

Commodity Accelerator (GPU)

InterconnectPCI-X 16 lane

64 Gb/s1 GW/s

6 GB

Future Computer Systems♦ Most likely be a hybrid design

Standard multicore chips and accelerator (GPUs)

♦ Today accelerators are attached♦ Next generation more integrated♦ Intel’s MIC architecture “Knights Corner”

48 x86 cores♦ AMD’s Fusion

Multicore with embedded graphics ATI♦ Nvidia’s Project Denver plans to develop

an integrated chip using ARM architecture

11

Performance Development in Top500

0.1

1

10

100

1000

10000

100000

000000

000000

000000

1E+09

1E+10

1E+11

1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020

1 Eflop/s

1 Gflop/s

1 Tflop/s

100 Mflop/s

100 Gflop/s

100 Tflop/s

10 Gflop/s

10 Tflop/s

1 Pflop/s

100 Pflop/s

10 Pflop/s

N=1

N=500

12

13

Major Changes to Software & Algorithms• Must rethink the design of our

algorithms and softwareAnother disruptive technology• Similar to what happened with cluster

computing and message passingRethink and rewrite the applications, algorithms, and softwareData movement is expenseFlop/s are cheap, so are provisioned in excess

Critical Issues at Peta & Exascale for Algorithm and Software Design• Synchronization-reducing algorithms

Break Fork-Join model

• Communication-reducing algorithmsUse methods which have lower bound on communication

• AutotuningToday’s machines are too complicated, build “smarts” into software to adapt to the hardware

• Fault resilient algorithmsImplement algorithms that can recover from failures/bit flips

• Reproducibility of resultsToday can’t guarantee this.

International Exascale Software Project

Attendees from universities, research institutes, government, funding agencies, research councils, hardware and software vendors, industry

Steering CommitteeJack Dongarra, U of Tennessee/Oak Ridge National Lab, US

Pete Beckman, Argonne Nat. Lab, US

Franck Cappello, INRIA, FR

Thom Dunning, NCSA, US

Thomas Lippert, Jülich Supercomputing Centre, DE

Satoshi Matsuoka, Tokyo Inst. of Tech, JP

Paul Messina, Argonne Nat. Lab, US

Patrick Aerts, Netherlands Organization for Scientific Research, NL

Anne Trefethen, Oxford, UK

Mateo Valero, Barcelona Supercomptuing Ceneter, Spain

16

International Exascale Software Project Objectives

To enable the international HPC community to improve, coordinate and leverage their collective investments and development efforts.To develop a plan for producing a software infrastructure capable of supporting exascale applications

Thorough assessment of needs, issues and strategiesDevelop a coordinated software roadmapProvide a framework for organizing the software research communityEngage vendors to coordinate on how to deal with anticipated scaleEncourage and facilitate collaboration in education and training

What Next? Moving from “What to Build” to “How to Build”

TechnologyDefining and developing the roadmap for software and algorithms on extreme-scale systemsAssessing the short-term, medium-term and long-term software and algorithm needs of applications for peta/exascale systems

www.exascale.org

What Next?Moving from “What to Build” to “How to Build”

OrganizationExploring ways for funding agencies to coordinate their support so that they complement each otherExploring how laboratories, universities, and vendors can work together on coordinated HPC softwareCreating a plan for working closely with HW vendors and application teams to co-design future architectures

www.exascale.org

What Next?Moving from “What to Build” to “How to Build”

ExecutionDeveloping a strategic plan for moving forwardCreating a realistic timeline for constructing key organizational structures and achieving initial goalsExploring community development techniques and risk plans to ensure key components are delivered on time

www.exascale.org

US TeraGrid

An instrument that delivers high-end IT resources/services

a computational facility – over two PFlopsScience Gateways –discipline-specific web-portal front-ends a data storage and management facility – 20 PetaBytesa high-bandwidth national data network

Support, education and training eventsAvailable freely to research and education projects with a US lead

TeraGrid Objectives

DEEP Science: enabling terascale and petascale sciencemake science more productive through an integrated set of very-high capability resources

address key challenges prioritized by users

WIDE Impact: empowering communitiesbring TeraGrid capabilities to the broad science community

partner with science community leaders

OPEN Infrastructure, OPEN Partnershipa coordinated, general purpose, reliable set of services and resources

partner with campuses and facilities

The eXtreme Digital (XD) Program

XD : third generation TeraGrid program

2002-2005: Distributed/Extended Terascale Facility2005-2011: Grid Infrastructure + Resource Providers2010-2016: eXtreme Digital (XD) + Service Providers

SDSC

TACC

UC/ANL

NCSA

ORNL

PU

IU

PSCNCAR

Caltech

USC/ISI

UNC/RENCI

UW

Resource Provider (RP)

Software Integration Partner

Grid Infrastructure Group (UChicago)

11 Resource Providers, One Facility

NICS

LONI

Network Hub

eXtreme Digital Resources

High-Performance Computing and Storage Services

High-Performance Remote Visualization and Data Analysis Services

2 awards; 5 years; $3M/yearIntegrating Services (5 years, $26M/year)

Coordination and Management Service (CMS)5 years; $12M/year

Technology Audit and Insertion Service (TAIS)5 years; $3M/year

Advanced User Support Service (AUSS)5 years; $8M/year

Training, Education and Outreach Service (TEOS)5 years, $3M/year

XSEDE : Governance

Leadershipled by NCSA, NICS, PSC, TACC and SDSC: centers with deep experiencepartners who strongly complement these centers with expertise in science, engineering, technology and education

Balanced governance modelstrong central management, rapid response to issues and opportunitiesdelegation and decentralization of decision-making authorityopenness to genuine stakeholder participation

stakeholder engagement, advisory committeesimproved professional project management practices

formal risk management and change control

25

XSEDE: Extending Impact

Coordinated national program with greater scope and scaleincreased diversity of topics, modes of delivery, and reach to new communities and audiencesbroaden participation among under-represented communities

Campus bridging for effective use of resourcesmore tightly integrate with campuses through expanded Champions program and additional bridging activities

Establish certificate and degree programsinstitutional incorporation of CS&E curricula; professional development certificateprepare undergraduates, graduates and future K-12 teachers

26

Summary HPC

www.exascale.org

Increasingly indispensably to scientific progress and economy competitivenessIndustrial competiveness ->time to marketNational securityQuality of human lifeKey element for the competiveness of knowledge based economiesNot HPC Leadership but innovation leadership