Implementation and experience with Big Red (a 30.7 TFLOPS IBM BladeCenter cluster), the

Data Capacitor, and HPSS

Craig A. Stewartstewart@iu.edu

1 November 2007

License Terms• Please cite this presentation as: Stewart, C.A. Implementation and experience with

Big Red (a 30.7 TFLOPS IBM BladeCenter cluster), the Data Capacitor, and HPSS). 2007. Presentation. Presented at: UITS Research Technologies Brownbag Lunch (Indianapolis and Bloomington, IN, 1 Nov 2007). Available from: http://hdl.handle.net/2022/14608

• Portions of this document that originated from sources outside IU are shown here and used by permission or under licenses indicated within this document.

• Items indicated with a © are under copyright and used here with permission. Such items may not be reused without permission from the holder of copyright except where license terms noted on a slide permit reuse.

• Except where otherwise noted, the contents of this presentation are copyright 2007 by the Trustees of Indiana University. This content is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.

Outline

• Brief history of implementation in TeraGrid and at IU

• System architecture• Performance analysis• User experience and science results• Lessons learned to date

Image from www.teragrid.org

IU & TeraGrid • IU: 2 core campuses, 6 regional campuses

• President-elect: Michael A. McRobbie

• Advanced computing: University Information Technology Services, Pervasive Technology Labs, School of Informatics

• Motivation for being part of TeraGrid:Support national research agendasImprove ability of IU researchers to use national cyberinfrastructureTestbed for IU computer science research

Big Red - Basics and history• IBM e1350 BladeCenter Cluster, SLES 9, MPICH,

Loadleveler, MOAB• Spring 2006: 17 days assembly at IBM facility,

disassembled, reassembled in 10 days at IU.• 20.48 TFLOPS peak theoretical, 15.04 achieved on

Linpack; 23rd on June 2006 Top500 List (IU’s highest listing to date).

• In production for local users on 22 August 2006, for TeraGrid users 1 October 2006

• Upgraded to 30.72 TFLOPS Spring 2008; ??? on June 2007 Top500 List

• Named after nickname for IU sports teams

Data Capacitor - Basics and History• Initially funded by $1.7M NSF grant to IU• Initially 535 TB of spinning disk - soon to

be expanded to more than 1 PB• Designed as a temporary holding place

for large data sets - a novel type of storage system

• Uses Lustre file system

HPSS - Basics and History

• High Performance Storage System• Designed initially by IBM and 5 DOE labs• IU has contributed code, remains the only

unclassified HPSS implementation with distributed storage

• Data written to HPSS is by default copied to IUB and IUPUI

Motivations and goals

• Initial goals for 20.48 TFLOPS system:

Local demand for cycles exceeded supply

TeraGrid Resource Partner commitments to meet

Support life science research

Support applications at 100s to 1000s of processors

• 2nd phase upgrade to 30.72 TFLOPS

Support economic development in State of Indiana

Why a PowerPC-based blade cluster?

• Processing power per node• Density, good power efficiency relative to

available processors

• Possibility of performance gains through use of Altivec unit & VMX instructions

• Blade architecture provides flexibility for future• Results of Request for Proposals process

Processor TFLOPS/ MWatt

MWatts/ PetaFLOPS

Intel Xeon 7041 145 6.88

AMD 219 4.57

PowerPC 970 MP (dual core) 200 5.00

Feature 20.4 TFLOPS 30.7 TFLOPS

Computational hardware, RAM

JS21 components Two 2.5 GHz PowerPC 970MP processors, 8 GB RAM, 73 GB SAS Drive, 40 GFLOPS

Same

No. of JS21 blades 512 768

No. of processors; cores 1,024 processors; 2,048 processor cores

1,536 processors; 3,072 processor cores

Total system memory 4 TB 6 TB

Disk storage

GPFS scratch space 266 TB Same

Lustre 535 TB Same

Home directory space 25 TB Same

Networks

Total outbound network bandwidth 40 Gbit/sec Same

Bisection bandwidth 64 GB/sec - Myrinet 2000 96 GB/sec - Myrinet 2000

Difference: 4 KB vs 16 MB page size

Linpack performance

Benchmark set

Nodes Peak Theoretical

TFLOPS

Achieved TFLOPS

%

HPCC 510 20.40 13.53 66.3

Top500 512 20.48 15.04 73.4

Top500 768 30.72 21.79 70.9

HPCC and Linpack Results (510 nodes)

G-HPL G-PTRANS

G-Random Access

G-FFTE EP-STREAM

Sys

EP-STREAM

Triad

EP-DGEMM

Random

Ring Bandwidth

Random Ring

Latency

GB/s usec

TFlop/s GB/s Gup/s GFlop/s GB/s GB/s GFlop/s

Total 13.53 40.76 0.2497 67.33 2468 17.73

Per processor

0.013264 0.0399 0.000244 0.066 2.42 8.27 0.0212

Data posted to http://icl.cs.utk.edu/hpcc/hpcc_results.cgi

April 20, 2023

20.4 TFLOPS e1350 (Big Red) vs. a 20.52 Cray XT3 at Oak Ridge National Labs, including 5200 single core 2.4 GHz AMD Opteron processors (left), and a 2.09 TFLOPS HP XC4000 owned by HP, Inc., including 256 dual-core ADM Opteron processors (right).

Elapsed time per simulation timestep among best in TeraGrid

Spinning Disk capacity growth

Bandwidth Challenge SC|2006

Alzheimer’s amyloid peptide analysis

CIMA - Crystallography Data Acquisition and Processing

Competition Performance

During testing4 x 2 trunked 1 Gb lines32 GB in 34 seconds - 941MB/s

CompetitionAll four experimentsSustained 5.5 - 6.6 Gb

HPSS I/O Speed Growth

• Simulation of TonB-dependent transporter (TBDT)

• Used systems at NCSA, IU, PSC

• Modeled mechanisms for allowing transport of molecules through cell membrane

• Work by Emad Tajkhorshid and James Gumbart, of University of Illinois Urbana-Champaign. Mechanics of Force Propagation in TonB-Dependent Outer Membrane Transport. Biophysical Journal 93:496-504 (2007)

• To view the results of the simulation, please go to: http://www.life.uiuc.edu/emad/ TonB-BtuB/btub-2.5Ans.mpgImage courtesy of Emad Tajkhorshid

ChemBioGrid• Analyzed 555,007

abstracts in PubMed in ~ 8,000 CPU hours

• Used OSCAR3 to find SMILES strings -> SDF format -> 3D structure (GAMESS) -> into Varuna database and then other applications

• “Calculate and look up” model for ChemBioGrid

WxChallenge (www.wxchallenge.com)• Over 1,000 undergraduate students, 64

teams, 56 institutions• Usage on Big Red:

~16,000 CPU hours on Big Red 63% of processing done on Big RedMost of the students who used Big Red

couldn’t tell you what it is• Integration of computation and data flows

via Lustre (Data Capacitor)

Overall user reactions• NAMD, WRF users very pleased• Porting from Intel instruction set a perceived and

sometimes real challenge in a cycle-rich environment

• MILC optimization with VMX not successful so far in eyes of user community

• Keys to biggest successes:Performance characteristics of JS21 nodesLinkage of computation and storage (Lustre -

Data Capacitor)Support for grid computing via TeraGrid

Evaluation of implementation• The manageability of the system is excellent• For a select group of applications, Big Red

provides excellent performance and reasonable scalability

• We are likely to expand bandwidth from Big Red to the rest of the IU cyberinfrastructure

• Quarry is a critical companion to Big Red; without Quarry Big Red would not be bnearly so successful

• Focus on data management and scalable computation critical to success

• Next steps: industrial partnerships and economic development in Indiana

Conclusions• A 20.4 TFLOPS system with “not the usual” processors was successfully

implemented serving local Indiana University researchers, and the national research audience via the TeraGrid

• Integration of computation and data management systems was critical to success• In the future Science Gateways will be increasingly important:

Most scientists can’t constantly chase after the fastest available system; gateway developers might be able to

Programmability of increasingly unusual architectures not likely to become easierFor applications with broad potential user bases, or extreme scalability on

specialized systems, Science Gateways will be critical in enabling transformational capabilities and supporting scientific workflows. Achieving broad use can only be achieved by relieving scientists of need to understand details of systems

Acknowledgements - Funding Sources

• IU’s involvement as a TeraGrid Resource Partner is supported in part by the National Science Foundation under Grants No. ACI-0338618l, OCI-0451237, OCI-0535258, and OCI-0504075

• The IU Data Capacitor is supported in part by the National Science Foundation under Grant No. CNS-0521433.

• This research was supported in part by the Indiana METACyt Initiative. The Indiana METACyt Initiative of Indiana University is supported in part by Lilly Endowment, Inc.

• This work was supported in part by Shared University Research grants from IBM, Inc. to Indiana University.

• The LEAD portal is developed under the leadership of IU Professors Dr. Dennis Gannon and Dr. Beth Plale, and supported by NSF grant 331480.

• The ChemBioGrid Portal is developed under the leadership of IU Professor Dr. Geoffrey C. Fox and Dr. Marlon Pierce and funded via the Pervasive Technology Labs (supported by the Lilly Endowment, Inc.) and the National Institutes of Health grant P20 HG003894-01

• Many of the ideas presented in this talk were developed under a Fulbright Senior Scholar ’s award to Stewart, funded by the US Department of State and the Technische Universitaet Dresden.

• Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF), National Institutes of Health (NIH), Lilly Endowment, Inc., or any other funding agency

Acknowledgements - People• Malinda Lingwall for editing, graphic layout, and managing process• Maria Morris contributed to the graphics used in this talk• Marcus Christie and Surresh Marru of the Extreme! Computing Lab contributed the LEAD

graphics• John Morris (www.editide.us) and Cairril Mills (Cairril.com Design & Marketing) contributed

graphics• Steve Simms - Data Capacitor project leadership and slides• Rick McMullen and all the Huffmans (CIMA)• Randy Bramley and Marie Ma (Obsidian)• Mookie Baik and Yogita Mantri (Chemistry)• Beth Plale, Dennis Gannon, AJ Ragusa, Suresh Marru, Chathura Herath (LEAD) • Maria Morris (Illustrator support)• Doug Balog, Derek Simmel (PSC)• Guido Juckeland (ZIH)• This work would not have been possible without the dedicated and expert efforts of the staff of

the Research Technologies Division of University Information Technology Services, the faculty and staff of the Pervasive Technology Labs, and the staff of UITS generally.

• Thanks to the faculty and staff with whom we collaborate locally at IU and globally (via the TeraGrid, and especially at Technische Universitaet Dresden)

Thank you

• Any questions?