uvic advanced networking day 28 november 2005 university of victoria research computing facility...

UVic Advanced Networking Day 28 November 2005

University of Victoria Research Computing Facility

Colin Leavett-Brown

UVic Advanced Networking Day 28 November 2005

Introduction

• Established in 1999

• Supports National Laboratories on Vancouver Island– NRC Herzberg Institute for Astrophysics

– NRCan Pacific Forestry Centre

– NRCan Pacific Geoscience Centre

• Broad range of research– Earth Science, Physical Sciences, Engineering

– Social Science, Medical Sciences

• Users across Canada, US and Europe

• Centre of Grid Activity (Forestry and Grid Canada)

UVic Advanced Networking Day 28 November 2005

Minerva HPC (1999 CFI Award)

• 8 node 8 way IBM SP

• Upgraded to 8 node 16 way in 2000

• 167th in the TOP 500 (highest ranking of any C3 facility?)

• Funded in part with an IBM Shared University Research (SUR) Award for $830,000

• Primary users: cosmology, climate simulations, engineering, and geosciences

• Still operational and utilized today.

UVic Advanced Networking Day 28 November 2005

Storage and Cluster (2001 CFI Award)

• 4 year project; online in March 2003

• Currently 80 TB disk and 440 TB tape capacity

• Primary users: Astronomy, Forestry, Particle Physics– Grid enabled for Forestry Data Grid

• IBM Beta Test Site for LTO2 (Jan-Apr 2003)

• Upgraded to LTO3 (March 2005)

UVic Advanced Networking Day 28 November 2005

Mercury & Mars Xeon Clusters

• Operational in March 2003

• 84 Dual Xeon blades (2.4 & 2.8 GHz)

• Top500 in 2003

• Now have 154 Dual Xeon blades (including 3.2GHz)

• Primary users: particle physics, engineering, geoscience

• Back-filled with particle physics simulations – 1 TB of data shipped to Stanford (SLAC) each week

• Grid-enabled and connected to Grid Canada– Particle physics and other simulation applications

– Goal is to back-fill other sites for simulation production

UVic Advanced Networking Day 28 November 2005

TAPOR Project (2001 CFI Award)

• TAPor (Text-Analysis Portal) project will help Canadian researchers in online textual research

• Six leading humanities computing centres in Canada, including UVic, will contribute collections of rare tapes and documents. – ISE, Robert Graves Diary, Cowichan Dictionary

– http://web.uvic.ca/hrd/tapor

• Database servers are integrated into the Storage Facility

UVic Advanced Networking Day 28 November 2005

Kikou Vector Computational Facility

UVic Advanced Networking Day 28 November 2005

Kikou Vector Computational Facility

• $2.42 M CFI Award (2004) for a Vector Computational Facility

• Purchased NEC-SX6– $5.3M

– Commissioned March 2005

– 4 Nodes, 32 Processors, 11.67 GFlops, 32/64GB, 10TB.

– Primary use for Climate Modelling (A.Weaver)

UVic Advanced Networking Day 28 November 2005

Llaima Opteron Cluster

• 2004 CFI/BCKDF awards.

• 40 Node, Dual Opteron (2.0 Ghz), 4GB.

• 20/40 Commissioned September 2005.

• Will have direct SAN attachment.

• Primary use is for Astronomy (S. Ellison)

UVic Advanced Networking Day 28 November 2005

Networking• Campus Networks are primarily optical networks with Cisco 6500

series routers.

• Connections to the access layer are generally twisted pair copper.

• Both a general use and a dedicated Research Network are provided (GE), interconnected via access layer.

• Research network used for I/O, node interconnect, and direct connections BCNet/CA*Net4.

UVic Advanced Networking Day 28 November 2005

Networking• External connections: commodity

network (100E), ORAN/CA*Net4 (optical & GE) through BCNet.

• CA*Net4/ to other national networks– Abilene, NLR, ESNET (US), GEANT

(EUR), JANET (UK), JAXA (JPN), KREONET (KOR), AARNET (AUS)

• Lightpaths near future (Jan 06)– ~1GB going to ~10GB

– Optical Switching: BCNet, Member Sites, TXs.

– Applications: HEPNet/C$ Testbed, CCCma, Babar/SLAC, ATLAS/TRIUMF, WG2

UVic Advanced Networking Day 28 November 2005

Infrastructure Challenges

• March 2002, 43KW

• Research Load:– March 2000, Minerva, 6KW

– March 2003, Storage, 30KW

– March 2003, Mercury & Mars, 40KW

– April 2004, TAPoR, 5kw

– March 2005, Kikou, 45 KW

– September 2005, Llaima, 20KW

• November 2005, 150KW

UVic Advanced Networking Day 28 November 2005

Infrastructure Challenges

• Ran out of space, power, & A/C.

• $700K renovation.

• Increased room by 32m².• Added 44 Tons of A/C.

• Added 225KW UPS.

• Still growing, time for another renovation.

UVic Advanced Networking Day 28 November 2005

Support

• Managed by the University of Victoria Computer Centre– Under the Research and Administrative Systems Support Group

– 3 FTE dedicated to the Research Computing

– Augmented by approximately 9 other sysadmins, network and hardware technicians

– 24/7 almost.

• C3 TASP Support– Position was used to assist with parallel programming support very

successfully.

– Position is currently vacant.

UVic Advanced Networking Day 28 November 2005

User community

• Minerva (~150 user accounts)– NRC, Queens, UBC, SFU, NRCan, Alberta, Washington,

Dalhousie

• Storage Facility– HIA and PFC use facility to distribute images to their user

community – TAPOR will provide a database service for social science

• Cluster (~60 user accounts not including Grid Canada)– NRC (Ottawa), NRCan (Geoscience and Forestry)– Montreal, UBC, TRIUMF, Alberta, NRC– Germany, UK, US (SLAC)

UVic Advanced Networking Day 28 November 2005

Plans for 2006• Storage

– Currently at the beginning of 4th year

– 200 TB disk and 1000 TB Tape

– Higher bandwidth, remove single points of failure.

– Minimise ongoing maintenance costs.

• Mercury Cluster– 50 blades per year

• Llaima– Commission remaining 20 nodes.

– Upgrade to multi-core processors?

• Kikou– Another 8-way node?

UVic Advanced Networking Day 28 November 2005

Future plans

• Next CFI competition– Renewal of Minerva HPC

– Continued support of Storage

– Modest expansion of the Mercury Cluster

– Upgrade of TAPoR

– Part of WG2

• Venus and Neptune Projects– $50 Million CFI projects to instrument the floor of the Pacific

shelf

– Venus (inland strait) will take data in 2005

– Neptune begins data taking in 2006-2007

UVic Advanced Networking Day 28 November 2005

Summary

• UVic RCF supports a broad range of users – A large fraction of the users are not part of the C3 community

• Wide range of computational and storage requirements– HPC, Cluster, Storage, Database, Grid

• Very successful– Operational very quickly and efficiently

– 2 machines in the TOP500

– Large storage repository

– Willingness to experiment: Beta test site, 2 Grid modes