state of hcc 2012 dr. david r. swanson director, holland computing center

State of HCC2012

Dr. David R. SwansonDirector, Holland Computing Center

Nature Communications, July 17, 2012

Nebraska Supercomputing Symposium 2012

HCC CPU Hour Usage 2012

Zeng (Quant Chem) 4.5M

Starace (AMO Phys) 2.7M

Rowe (Climate) 2.0M

NanoScience 6.4M

CComp Bio 3.0M B

Comp Sci 1.7M C

Physics 0.7M

Mech E 0.4M

High Performance Computing

• http://t2.unl.edu/status/hcc-status• Xiao Zeng, Chemistry, UNL (prior slide)• DFT and Car Parrinello MD• HPC – tightly coupled codes• Requires expensive low-latency local network

(infiniband)• Requires high-performance storage (Panasas,

Lustre) • Requires highly reliable hardware

Eureka! A Higgs! (or at least something currently indistinguishable)

• "I think we have it. We have discovered a particle that is consistent with a Higgs boson." – CERN Director-General Rolf Heuer

US CMS Tier2 Computing

Compact Muon Solenoid (CMS)

5.5 mi

Large Hadron Collider

CMS Grid Computing Model

Eureka! A Higgs! (or at least something currently indistinguishable)

• Ca. 50 PB of CMS data in entirety• Over 1 PB currently at HCC’s “Tier2”, 3500

cores• Collaboration at many scales

– HCC and Physics Department– Over 2700 scientists worldwide– International Grid Computing Infrastructure– Data grid as well– UNL closely linked to KU, KSU physicists via

a jointly hosted “Tier3” Nebraska Supercomputing Symposium 2012

Data Intensive HTC

• Huge database• Requires expensive high-bandwidth wide area

network (dwdm fiber)• Requires high-capacity storage (HDFS, dCache) • HTC – loosely coupled codes• Requires hardware

Outline

• HCC Overview• New User report• HCC-Go• Moving Forward (after break)

– Next purchase– It’s the Data, stupid… – Other Issues

Outline

• New User report• HCC-Go• Moving Forward (next section)

– Next purchase (motivation)– New Communities– PIVOT– It’s the Data, stupid…

HOLLAND COMPUTING CENTER OVERVIEW

HCC @ NU

• Holland Computing Center has a University-wide mission to – Facilitate and perform computational and

data intensive research– Engage and train NU researchers, students,

and other state communities

– This includes you! – HCC would be delighted to collaborate

Computational Science – 3rd Pillar

Experim

Theory

Computation/D

Lincoln Resources

• 10 staff• Red• Sandhills• 5,000 compute

cores• 3 PetaBytes

storage in HDFS

Sandhills “Condominium Cluster”

• 44 nodes X 32-core, 128 GB, IB

• Lustre (175 TB)• Priority Access

– $HW + $50/month– 4 groups currently

• SLURM

Omaha Resources

• 3 Staff • Firefly • Tusker• 10,000 compute

cores• 500 TB storage• New offices soon:

158J PKI

Tusker

• 106*64= 6784 cores

• 256 GB/node• 2 nodes w/ 512

GB• 360 TB Lustre

– 100 TB more en route

• QDR IB• 43 TFlop

Tusker

• ¼ footprint of Firefly

• ¼ the power• 2X the TFLOPS• 2X the storage• Fully utilized• Maui/Torque

In between …

• HCC (UNL) to Internet2: 10 gbps• HCC (Schorr) to HCC (PKI): 20 gbps• Allows us to do some interesting things

– “overflow” jobs to/from Red– DYNES project– Xrootd mechanism

HCC Staff

• HPC Applications Specialists– Dr. Adam Caprez– Dr. Ashu Guru– Dr. Jun Wang– Dr. Nicholas

Palermo

• System Administrators– Dr. Carl Lundstedt– Garhan Attebury– Tom Harvill– John Thiltges– Josh Samuelson– Dr. Brad Hurst

HCC Staff

• Other Staff– Dr. Brian

Bockelman– Joyce Young

• GRAs– Derek Weitzel– Chen He– Kartik Vedalaveni– Zhe Zhang

• Undergraduates– Carson Crawford– Kirk Miller– Avi Knecht– Phil Brown– Slav Ketsman– Nicholas Nachtigal– Charles Cihacek

HCC Campus Grid

• Holland Computing Center resources are combined into an HTC campus grid– 10,000 cores, 500 TB in Omaha– 5,000 cores, 3 PB in Lincoln– All tied together via a single submission

protocol using OSG software stack– Straightforward to expand to OSG sites

across the country, as well as to EC2 (cloud)– HPC jobs get priority; HTC ensures high

utilizationNebraska Supercomputing Symposium 2012

HCC Model for a Campus Grid

Me, my friends and everyone else

Campus

HCC & Open Science Grid

• National, distributed computing partnership for data-intensive research– Opportunistic computing– Over 100,000 cores– Supports the LHC experiments, other

science– Funded for 5 more years– Over 100 sites in the Americas– Ongoing support for 2.5 (+3) FTE at HCC

It Works!

HCC Networking Monitoring

OSG Resources

Working philosophy

• Use what we buy– These pieces of infrastructure are linked, but improve

asynchronously – Depreciation is immediate– Leasing is still more expensive (for now)– Buying at fixed intervals mitigates risk, increases ROI– Space, Power and Cooling have a longer life span

• Share what we aren’t using– Share opportunistically – retain local ownership– Consume opportunistically – there is more to gain!– Collaborators, not just consumers– Greater good vs. squandered opportunity

Working philosophy

• A Data deluge is upon us• Support is essential

– If you only build it, they still may not come– Build incrementally and buy time for user training– Support can grow more gradually than hardware

• Links to national and regional infrastructure are critical – Open Source Community– GPN access to Internet2– Access to OSG, XSEDE resources– Collaborations with fellow OSG experts– LHC

HCC New Users

FY UNL-City

UNL-East

UNO UNMC Outside NU system

2011 424 (74) 33 (10) 75 (19) 30 (17)

112 (26)

2012 519 (95) 50 (17)

105 (30) 35 (5)

130 (18)

New User Communities

• Theatre, Fine Arts/Digital Media, Architecture• Psychology, Finance

• UNMC

• Puerto Rico

• PIVOT collaborators

HCC NEW USER REPORT:HEATH ROEHR

HCC-GO :DR. ASHU GURU

MOVING FORWARD

NEW PURCHASE

$2M for …

• More computing– need ca. 100 TF to hit Top500 for Jun 2013 – Likely use all of funds to hit that amount

• More storage– Near-line archive (9 PB)– HDFS

• Specialty hardware– GPGPU/Viz– Mic hardware

More computing

• How much RAM/core? • Currently almost always oversubscribed• Large scale jobs almost impossible (> 2000

core)• Safest investment – will use right away• Firefly due to be retired soon – EOL

More computing

More Computing

More storage

• Most rapidly growing demand• Growing contention, can’t just queue up• Largest unmet need (?)

Storage for $2M

• $2M HDFS cluster – 250 nodes– 4000 cores (Intel)– 9.0 PB (RAW)– 128 GB / node

Other options

• GPGPUs most Green option for computing• Highest upside for raw power (Top500)• Mic even compatible with x86 codes

• SMP uniquely meets some needs, easiest to use/program

• Bluegene, Tape silo, …

HCC personnel timeline

1999 2002 2005 2009 2012

Personnel 2 3 5 9 13

HCC Personnel Numbers

HCC networking timeline

1999 2002 2005 2009 2012

WAN B/W 0.155 0.155 0.622 10 30

HCC WAN Bandwidth

HCC cpu timeline

1999 2002 2005 2009 2012

CPU Cores 16 256 656 6956 14492

HCC CPU Cores

HCC storage timeline

1999 2002 2005 2009 2012

Capacity 0.108 1.2 31.2 1200 3250

HCC Storage Capacity (RAW)

30,000X

Composite Timeline

• Data increase/ CPU Cores = 33• Data increase/ WAN bandwidth = 150• It takes a month to move 3 PB at 10 Gb/sec

• Power < 100X increase, largely constant last 3 years

Storage at HCC

• Affordable, Reliable, High Performance, High Capacity– Pick 2 – So multiple options

• /home• /work• /shared• Currently, no /archive

• Reliable• Low performance

– No W from workers• ZFS• Rsync’ed pair, one in Omaha, one in Lincoln• Backed up incrementally, requires severe

quotas

• High performance• High(er) capacity• Not permanent storage• Lenient quotas• More robust, more reliable “scratch space”• Subject to purge as needed

/share

• Purchased by given group• Exported to both Lincoln and Omaha machines• Usually for capacity, striped for some reliability

Storage Strategy

• Maintain /home for precious files– Could be global

• Maintain /work for runtime needs– Remain local to cluster

• Create /share for near-line archive– 3-5 year time frame (or less)– Use for accumulating intermediate data,

then purge– Global access

Storage strategy

• Permanent archival has 3 options– 1) library– 2) Amazon glacier

• Currently $120/TB/year– 3) tape system

HCC Data Visualizations

• Fish!• HadoopViz• OSG Google Earth

• Web-based monitoring– http://t2.unl.edu/status/hcc-status/– http://hcc.unl.edu/gratia/index.php

state of hcc 2012 dr. david r. swanson director, holland computing center

pb of cms data

nuholland computing

data intensive researchengage

gb360 tb lustre

higgs boson

compact muon solenoid

computational science

hcc cpu hour usage

Documents

rod swanson resume

hcc overview

swanson photography

i ntroducing s tatistical i nference using r andomization m...

hcc lesson6

(swanson 1961) swanson magnus effect review paper

sayre chpt 11 painting hcc aa 1 - hcc learning web

kickstart 2018 - hcc · 2019-05-29 · kickstart 2018 david...

liability - tokio marine hcc...tokio marine hcc is a trading...

fall - hcc

swanson flo performance

cancer hcc

hcc embolization

adam swanson

hcc talkppt

hcc presentation

an active approach to statistical inference using...

2018 unit total development cost (tdc) limits · hcc hcc...

hcc transformation journey: hcc presentation at acct 2016

hcc (final)