msu supercomputing facilities

24
1

Upload: peter-bryzgalov

Post on 30-Mar-2016

222 views

Category:

Documents


0 download

DESCRIPTION

Moscow State University supercomputing facilities. One of 4 pamphlets designed at MSU Research computing center for International Supercomputing Conference (ISC) 2011 at Hamburg.

TRANSCRIPT

Page 1: MSU supercomputing facilities

1

Page 2: MSU supercomputing facilities

2

Page 3: MSU supercomputing facilities

3

Contents

Preface 4

Computers and computing in Moscow State University 5

MSU supercomputers: “Lomonosov” 12

MSU supercomputers: SKIF MSU “Chebyshev” 16

MSU supercomputers: IBM Blue Gene/P 18

MSU supercomputers: Hewlett-Packard “GraphIT!” 20

Perspective supercomputing technology: reconfigurable supercomputers 22

Page 4: MSU supercomputing facilities

4

The history of computers at Moscow State Univer-sity goes back to the mid-fifties of the 20th century when Research Computing Center of Moscow State University was founded in 1955 and equipped with up-to-date computing hardware. This made it possible for university researchers to solve many challenging problems in meteorology, satellite and manned space flights, aerodynamics, structural analysis, mathematical economy, and other fields of science. Between 1955 and the early 1990s, more than 25 mainframe computers of various architec-ture and performance were installed and actively used at Moscow State University.

Since the end of the 1990s, Moscow State University has begun to exploit high-performance comput-ing systems based on cluster technologies. The first high-performance cluster installed at Moscow State University in 1999 was the first one in Rus-sian education and science institutions. It was able to perform 18 billion operations per second. Now, Moscow State University Supercomputing Center has two systems included in the Top500 list: “Lo-monosov” and SKIF MSU “Chebyshev”. Other MSU

supercomputers are IBM Blue Gene/P, Hewlett-Packard “GraphIT!” and FPGA-based RVS-5. The major computing facility of the Center is the “Lo-monosov” supercomputer with a recently increased peak performance up to 1.3 PFlops.

Today more than 500 scientific groups from Moscow State University, institutes of the Russian Academy of Sciences, and other educational and scientific organizations of Russia are the users of Moscow University Supercomputing Center. The main areas of fundamental research with super-computer applications are magnetohydrodynam-ics, quantum chemistry, seismology, drug design, geology, material science, global climatic changes, nanotechnology, cryptography, bioinformatics, bioengineering, astronomy, etc. In recent years, the range of supercomputer applications has expanded incredibly and Moscow State University is looking forward to reach exaflops frontiers.

Preface

Page 5: MSU supercomputing facilities

5

Computers and computing in Moscow State University

In 1956, Research Computing Center (RCC) of Moscow State University received its first computer “Strela”. It was the first serially manufactured main-frame in the USSR. A total of seven mainframes were produced, the one supplied to RCC had number 4. “Strela” mainframe functioned with a three-address instruction set capable of implement-

ing approximately 2000 operations per second. It had a clock cycle of 500 microseconds, RAM of 2048 words with 43 bits each, energy consumption of 150 KW. The computer occupied up to 300 square meters.

Page 6: MSU supercomputing facilities

6

Computer “Setun” was designed in RCC, with N.P. Brusentsov as a chief designer. In 1959, RCC launched “Setun” prototype and in 1961 “Setun” started to be manufactured serially. It was an im-pressive and extraordinary computer, being the first one in the world that was based on ternary, not bi-nary, logic. Trit, having capacity superior to that of

a bit, can exist not in two, but in three states: 1,0,-1. The “Setun” computer took up to 25-30 square me-ters, and required no special cooling. Its frequency was 200 kHz. Fifty computers were produced from 1961 to 1965.

Page 7: MSU supercomputing facilities

7

In May 1961, M-20 computer was installed in RCC. It’s worth mentioning, that mainframes of “M” series (М-20, М-220, М-222), built under the supervision of distinguished academician S.A. Leb-edev, were widely-spread in the USSR. Mainframe M-20 provided 20000 operations per second. It had ferrite core-based RAM with capacity of 4096 words, with external memory stored on drums and magnetic tapes. These common and efficient mainframes had essential influence on the develop-ment of computational mathematics in the former Soviet Union. For instance, a block method for

solving complicated algebraic problems that al-lowed dealing with systems of any rank and used only 300 words of RAM was developed specially for these mainframes. Using this method both matrix and system’s right-hand side vector fit into a slow memory but nevertheless problems were solved almost as fast as if all data were stored in RAM. The programs based on this technology were rather efficient: it took only 9 minutes to solve algebraic systems of rank 200 on M-20.

BESM-4 computer became a part of RCC com-putational facilities in 1966. BESM-4 ferrite cores memory capacity varied from 4096 to 8192 words with 45 bits each. Numbers were represented in floating-point mode in binary system, while the range of absolute values was from 2-63 to 263. Its memory cycle was 10 microseconds, total storage space on drum memory was 65536 words (4 drums of 16384 words each), external memory capacity via magnetic tapes contained 8 blocks of 2 million words each. BESM-4 occupied three cabinets using 65 square meters. It required 8 kW for functioning and had an automatic internal air cooling system.

Page 8: MSU supercomputing facilities

8

BESM-6 computer was and is still considered to be of great importance to Russian history of computer development. The chief designer of this model was again S.A. Lebedev. Designing of BESM-6 was com-pleted in 1967 and its serial production was started in 1968. Same year RCC received its first BESM-6 computer, and despite its serial number 13 it proved to be lucky for the Center. As a result RCC installed its second BESM-6 computer in 1975, and then the third and the forth ones in 1979. During this period total number of 355 BESM-6 mainframes was pro-duced in the USSR.

Parallel processing of computer instructions was widely used in the architecture of BESM-6 com-puter: simultaneously 14 single-address instructions

being at different stages could be processed. Buffers for intermediate storage of instructions and data allowed three subsystems of RAM modules, control and arithmetic units to work in parallel and asyn-chronously. Content-addressable memory on fast registers (a predecessor of cache memory) allowed this computer to memorize most frequently used operands and thus to decrease a number of refer-ences to RAM. Interleaving RAM allowed simul-taneous access to separate modules of RAM from different parts of mainframe.

BESM-6 had RAM on ferrite cores capable of storing 32 000 of 50-bit words. This number was later increased to 128 000 words. The BESM-6 peak performance was one million instructions per

Page 9: MSU supercomputing facilities

9

second. The computer had about 60000 transistors and three times more diodes. It had a frequency of 10 MHz, occupied up to 150-200 square meters and consumed 30 KW of energy supply.

RCC has also used mainframes from other series. In 1981, along with four BESM-6 mainframes RCC

was equipped with two ES-1022, two MIR-2 and MINSK-32 computers. In 1984, two-processor ES-1045 was installed. Since 1986, RCC has used a series of minicomputers: SM-3, SM-4 and SM-1420.

Page 10: MSU supercomputing facilities

10

Since 1999, Research Computing Center has decid-ed to focus its main attention on cluster supercom-puters. The result of this decision wasn’t obvious at that time, but later it has proved to be the right one. The first cluster consisted of 18 compute nodes con-nected via a high-speed SCI network. Each node contained two Intel Pentium III/500 MHz proces-sors, 1 GB of RAM and a 3.2 GB HDD. The system peak performance was 18 GFlops. The SCI network with a high data transfer rate (80 MB/s) and low latency (5.5 ns) made this system very effective for solving a wide range of problems. Research groups formed around the first cluster started using a new type of technology – parallel computers with dis-tributed memory in order to boost their research.

In 2002, the second cluster with a standard low-cost and effective Fast Ethernet technology for com-munication and control was installed. This cluster contained 20 nodes of one type (2 x Intel Pentium III/850 MHz, 1 GB, 2 x HDD 15 GB) along with 24 nodes of another type (2 x Intel Pentium III/1 GHz, 1 GB, HDD 20 GB). With a total number of 88 pro-cessors, it had peak performance of 82 GFlops.

In 2004, in the frame of a joint project of three departments of Moscow State University (Research Computing Center, Skobeltsyn Institute of Nuclear Physics and Faculty of Computational Mathematics and Cybernetics) new data storage was installed. It included Hewlett-Packard XP-1024 disk array along with an automated tape library Hewlett-Packard

Page 11: MSU supercomputing facilities

11

ESL 9595 with a total capacity of 40 TB. In the same year a new Hewlett-Packard cluster with 160 AMD Opteron 2.2 GHz processors and a new InfiniBand network technology was launched in the super-computing center. This cluster peak performance exceeded 700 GFlops. By that time more than 50 research groups from MSU, Russian Academy of Sciences and other Russian universities had become active users of MSU supercomputing facilities.

Now Moscow State University Supercomputing Center exploits “Lomonosov”, SKIF MSU “Che-byshev”, “GraphIT!”, IBM Blue Gene/P super-computers and several small HPC clusters, with a peak performance of the “Lomonosov” flagship at 1.3 PFlops. Taking the supercomputing road more than ten years ago Moscow State University Super-computing Center is planning to move forward to exaflops and further in the future.

Page 12: MSU supercomputing facilities

12

MSU supercomputers: “Lomonosov”Moscow State University hosts a number of HPC systems. SKIF MSU “Chebyshev” supercomputer has been the most powerful one until recently. This 60 TFlops supercomputer was installed in 2008; and after deployment it became very soon clear that the demand for computing power far exceeded its capabilities. By 2009 a significant expansion of MSU supercomputing facilities had become

a necessity and MSU decided to acquire a new, much more powerful system enabling research-ers to expand computations and to perform more accurate simulations. It became evident that the new supercomputer would have to contribute to the growth of Russia’s overall competitiveness by foster-ing discoveries and innovations in leading research centers of the country.

Page 13: MSU supercomputing facilities

13

Robust price/performance, scalability, and fault tolerance were the key requirements to the new system. “Lomonosov” supercomputer delivered by the Russian company T-Platforms currently has 1.3 PFlops peak performance.

“Lomonosov” is divided into 2 partitions by nodes architecture: x86 part with peak performance of 510 TFlops and GPU part with peak performance of 863 TFlops. In general, “Lomonosov” uses 6 types of compute nodes and incorporates processors of different architecture. The resulting hybrid instal-lation has enough flexibility for enabling optimum performance for a wide range of applications.

The primary compute nodes generating over 94% of x86 part performance are based on T-Platforms T-Blade2 system. Using six-core Intel Xeon X5670 Westmere processors, T-Blade2 brings up to 27 TFlops of compute power in a standard 42U rack. “Lomonosov” also contains a number of T-Blade 1.1 compute nodes with increased amount of RAM and local disk storage for memory-intensive applica-tions. The 3rd type of compute nodes is based on T-Platforms PeakCell S platform using PowerXCell 8i processors.

GPU part of “Lomonosov” supercomputer is based on the next generation of T-Platforms blade systems TB2-TL. The TB2-TL system is based on the newest

Page 14: MSU supercomputing facilities

14

TL-blade design. With 16 TL blades, it packs 32 Tesla X2070 GPUs and 32 Intel Xeon 5630 CPUs to deliver 17.8TF of peak DP performance per single TB2 enclosure. With 6 TB2-TL systems installed into a 42U rack cabinet, total performance of 106.6 TFlops per rack is reached.

“Lomonosov” uses 40 Gb/s QDR Infiniband tech-nology as a primary interconnect. To ensure fast data transfer and to reduce network congestion, T-Blade2 chassis incorporates excess InfiniBand external ports, providing impressive 1.6 TB/s of the overall external bandwidth of QDR InfiniBand integrated switches. The dedicated global barrier network of T-Blade2 allows fast synchronization of computing jobs running on separate nodes, while the global interrupt network significantly reduces the influence of OS jitter by synchronizing the pro-cess scheduling over the entire system. As a result, processors communicate much more efficiently, enabling high scalability of the most demanding parallel applications.

The supercomputer uses 3-level storage system:

• 500 TB of T-Platforms ReadyStorage SAN 7998 external storage with Lustre parallel file system. The solution enables parallel access of compute nodes to data with sustained aggregated read throughput of 30 GB/s and sustained aggre-gated write throughput of 24 GB/s;

• 300 TB high availability NAS storage for users home directories;

• 1 PB tape library with hierarchical storage software.

A very high degree of fault tolerance is a necessity for installations of such scale. To this end, redun-dancy of all critical subsystems and components was implemented – from cooling fans and power supplies on compute nodes to the entire engineer-ing infrastructure. To ensure even greater reliability, primary compute nodes have neither hard discs nor cables inside the chassis, and contain a number of special hardware features such as fault-tolerant memory module slots.

Page 15: MSU supercomputing facilities

15

Peak performance 1373 TFlops

Linpack performance 674 TFlops

Linpack efficiency 49%

Primary / secondary compute nodes T-Blade2, TB2-TL / T-Blade1.1, PeakCell S

4-core Intel Хеоn 5570 2.93 GHz CPUs 8 840

6-core Intel Xeon 5670 2.93 GHz CPUs 1 360

4-core Intel Xeon 5630 2.53 GHz CPUs 1 554

NVIDIA X2070 GPUs 1 554

Other processor types PowerXCell 8i

Total RAM 85 TB

Total number of cores 94 172

Primary / secondary interconnect QDR Infiniband 4x / 10G Ethernet, Gigabit

Ethernet

External storage 3-level storage:

•500 TB T-Platforms ReadyStorage SAN 7998/Lustre;

•300 TB NAS storage;

•1 PB tape library

Operating system Clustrx T-Platforms Edition

Total area (supercomputer) 252 m2

Power consumption 2.8 MW

“Lomonosov”

Page 16: MSU supercomputing facilities

16

MSU supercomputers: SKIF MSU “Chebyshev”On March 19, 2008 Moscow State University, T- Platforms company, Program Systems Institute of Russian Academy of Sciences and Intel Corporation announced the deployment of the most powerful supercomputer in Russia, CIS and Eastern Eu-rope SKIF MSU “Chebyshev” that was built in the framework of the supercomputer program “SKIF-GRID” sponsored by the Union State of Russia and Belarus. The peak performance of the supercom-puter based on 1 250 Intel Xeon E5472 quad-core processors, is 60 TFlops. The Linpack performance of 47.17 TFlops (78.6% of peak performance) had become the best efficiency result among all quad-core Xeon-based systems in the top hundred of the June 2008 edition of the Top500 list where SKIF MSU “Chebyshev” was ranked №36. It was ranked №5 in the recent (March 2011) edition of Top50 rat-ing list of the most powerful supercomputers in the Commonwealth of Independent States.

The supercomputer is based on T-Blade modules developed by T-Platforms. T-Blade incorporates up to 20 Intel Xeon quad-core processors (3.0 GHz, 45 nm) in a 5U enclosure, which at the moment of system delivery provided the best computing den-sity among all Intel-based blade solutions presented on the market. The system network is based on the DDR InfiniBand technology with Mellanox 4th generation microchips.

The T-Platforms ReadyStorage ActiveScale Clus-ter storage system specifically designed for Linux clusters provides direct parallel access to data for all compute nodes eliminating bottlenecks of traditional network storage. Data storage capacity of SKIF MSU “Chebyshev” is 60 TB. The unique feature of the T-Platforms ReadyStorage ActiveScale Cluster system is its scalability: when new storage modules are added, not only storage capacity but also the overall network performance is increased.

Page 17: MSU supercomputing facilities

17

Peak performance 60 TFlops

Linpack performance 47 TFlops

Linpack efficiency 78.6%

Compute racks / total racks 14 / 42

Blade enclosure / blade nodes 63 / 625

Number of CPUS / cores 1 250 / 5 000

Processor type 4-core Intel Хеоn 5472 3.0 GHz

Total RAM 5.5 TB

Primary / secondary interconnect DDR Infiniband / Gigabit Ethernet

Power consumption 330 KW

Top500 position 36 (2008.VI)

SKIF MSU “Chebyshev”

Page 18: MSU supercomputing facilities

18

MSU supercomputers: IBM Blue Gene/PSince 2008 the IBM Blue Gene/P supercomputer has been operating at the Faculty of Computational Mathematics and Cybernetics of MSU. The MSU Blue Gene/P computer was one of the first systems of this series in the world. Blue Gene architecture has been developed by IBM in the framework of the project seeking for new solutions in high-performance computing. MSU Blue Gene/P was at the 128-th place in the Top500 issued in November 2008. It was ranked #15 in the March 2011 Top50 list of the CIS most powerful supercomputers.

The IBM Blue Gene/P system is a representative of a supercomputer family providing high performance, scalability, and facility to process large datasets and at the same time consuming significantly less

energy and space in comparison with the earlier systems.

The configuration of MSU Blue Gene/P includes two racks, containing totally 2 048 compute nodes, each consisting of 4 PowerPC 450 cores, working at 850 MHz frequency. The peak performance of the system is 27.9 TFlops.

The Blue Gene/P architecture has been developed for programs that scale well up to hundreds and thousands of processes. Individual cores work at a relatively low frequency, but applications being able to effectively use large numbers of processor units demonstrate higher performance as compared to many others supercomputers.

Page 19: MSU supercomputing facilities

19

Peak performance 27.9 TFlops

Linpack performance 23.9 TFlops

Number of racks 2

Number of compute nodes / I/O nodes 2 048 / 32

CPU model 4-core PowerPC 850 MHz

Number of CPUs / cores 2 048 / 8 192

Total RAM 4 TB

Programming technologies MPI, OpenMP/pthreads, POSIX I/O

Performance per watt 372 MFlops/W

Top500 position 128 (2008.XI)

IBM Blue Gene/P

Page 20: MSU supercomputing facilities

20

MSU supercomputers: Hewlett-Packard “GraphIT!”“GraphIT!” is the first cluster of MSU Super-computing Center based on GPU, an innovative supercomputing architecture. GPUs, originally designed for real-time 3D graphics acceleration, are now widely used to accelerate HPC. Compared to traditional CPUs, GPUs provide higher parallelism, higher FLops and memory bandwidth per chip, and also have higher cost- and energy-efficiency.

Hewlett-Packard “GraphIT!”Peak performance (CPU / GPU / CPU+GPU) 2.04 / 24.72 /26.76 TFlops

Linpack performance 11.98 TFlops

Racks / compute nodes 2 / 16

Node type DL380G6

Number of 6-cores Intel Xeon X5650 CPUs 32

CPUs per node 2

Number of GPUs 48

GPU type Nvidia«Fermi»TeslaM2050

Total CPU RAM / GPU RAM 768 GB / 144 GB

Per node CPU RAM / GPU RAM 48 GB / 9 GB

Data storage capacity 12 ТB

Primary / secondary interconnect QDR Infiniband 4x / Gigabit Ethernet

Power consumption 22 KW

“GraphIT!” was originally envisioned as a pilot GPU-based cluster which can be used as a testbed for practicing with hybrid programming tech-nologies. It was required to be small enough to fit into existing server room but powerful enough to be used for real-world applications. As a result, configuration based on 4 HP S6500 4U chassis, oc-cupying a total of 2 racks was chosen. Each chassis

Page 21: MSU supercomputing facilities

21

has 4 nodes, and each node has 3 NVidia “Fermi” Tesla M2050 CUDA-enabled GPUs, for a total of 16 compute nodes and 48 GPUs in the cluster. All compute nodes are connected by a high-speed 4x QDR InfiniBand network. This provides a total per-formance of 26.76 TFlops, of which 24.72 TFlops, or more than 92%, are due to GPU. It achieves Linpack performance of 11.98 TFlops, with 44% efficiency.

“GraphIT!” cluster is used to solve problems on mo-lecular dynamics, cryptoanalysis, quantum physics, climate modeling, as well as other computationally intensive problems which benefit from GPU usage. It is used by researchers from various MSU depart-ments as well as other research institutions.

Page 22: MSU supercomputing facilities

22

Perspective supercomputing technology: reconfigurable supercomputers

Reconfigurable supercomputer RVS-5 installed in Research Computing Center of MSU is one of the most powerful reconfigurable computing systems in the world. This system was designed in Research Institute of Multiprocessor Computing Systems, Southern Federal University (Taganrog, Russia). The heads of the design team were Prof. I. Kaliaev and Dr. I. Levin.

Page 23: MSU supercomputing facilities

23

“RVS-5” FPGA system

Base Module Features

FPGA model Xilinx Virtex-5

Number of racks 5

Number of FPGAs (11 mil. gates) 1 280

Total size of dynamic memory 100 GB

Power consumption 24 KW

Number of processor elements 512

Memory size 2 GB

Performance, SP (DP) 200 (100) GFlops

Board frequency 330 MHz

Frequency of information exchange 1200 MHz

Size 6U

Power consumption 190 W

sibility of using a large number of FPGAs for any program (all FPGAs of a rack).

Various scientific applications have been success-fully implemented on RVS-5. Among them are:

• Tomographic researches of near-surface layers of the Earth using acoustic and electromagnetic waves;

• Modeling and forecasting the hydrophysical and biogeochemical processes in the Sea of Azov;

• Modeling natural objects and processes in the functioning area of the Rostov atomic power station;

• Modeling astrophysical processes and adjust-ment of instrumental distortion of optical images;

• Creation of fundamentally new drugs and new generation materials.

The main computational element of the RVS-5 computer is a base module Alkor. Each Alkor mod-ule contains 16 FPGA Xilinx Virtex-5 chips. Base modules are connected together via LVDS channels which allow several base modules to be effectively assigned to a program. Four base modules form a computational block, four blocks per each rack.

Reconfigurable computing system RVS-5 outper-forms all known general purpose FPGA-based computing systems. Most programs for this su-percomputer are written in the high-level Colamo language, which has been created by developers of RVS-5. The main features of this language are high efficiency of programs written in Colamo and pos-

Page 24: MSU supercomputing facilities