a glimpse of ncsa’s role in support of research

National Center for Supercomputing Applications

A Glimpse of NCSA’s Role in Support of Research

Radha NandkumarProgram Director, International Affiliations & Campus

RelationsNational Center for Supercomputing Applications

[email protected]

U.S.- South America Workshop: Mechanics and Advanced Materials – Research and Education

August 2-6, 2004


Begin with Thanks

• Organizers – Profs. Borges, Dumont, Espinosa, Paulino, and Rochinha– For the invitation– Regular engagements– Making everyone feel welcome, special, and important

• NSF– Vision, encouragement and funding

• Other Brazilian Collaborators – Bruno Schulze, LNCC– Vinod Robello and Christina Maria Boeres - UFF– for their invitations, hosting the visit, and for several of their visits to NCSA to seal

our collaborations.• Brazilian Colleagues and Community

– Hospitality and warmth• Other particiapnts in this workshop for interactions and continued

discussions

• Added bonus and a sign of success - Events such as this workshop are also enabling newer collaborations “between” U.S. researchers


A little bit about myself

• Professional preparation– Nearly 2 decades of experience in HPC and computational

science (NCSA); among the staff since inception!– Completed recently (May ’02) an Executive MBA– Ph.D. from the University of Illinois, Urbana-Champaign

• Condensed Matter Physics in Astrophysical Systems (neutron stars) – Thesis Advisor – Prof. David Pines

– Observational X-ray Astronomy (in India) and Cosmic ray physics (Univ. of Chicago) prior to the above.

• Current Activities– Enabling International partnerships and collaborations for NCSA– Enabling and monitoring interdisciplinary computational science

research at NCSA


Presentation Outline

• Introduction to NCSA• Partnerships• Recent Changes &Trends • Computing Infrastructure• Software Infrastructure• Sample Applications



• NCSA– a unit of the University of Illinois at

Urbana-Champaign– a U.S. NSF HPC center started in

1985 with international collaboration, in its 19th year

• federal, state, university, and industry funded

– Transition from SCC & HPCC (12 yrs) to PACI Center (7 yrs) to SCI Center (soon)

– a globally recognized leader in HEC and computational science, scientific visualizations, innovative software

• Mission– providing access to leading computing

and information technologies • universities and industry


Strength of Innovation

NCSA TelnetMosaic

Scientific VisualizationsVirtual Director

HDF, D2K, Open source software, CAVElib“In-a-Box” Software Suite

Grid in a Box => NMIHigh End Computing Center – #4 in Top500

LES for Alliance and the TeraGrid


NCSA Personnel in a nutshell

• Current Direction & Vision– Rob Pennington, Acting Interim Director – Danny Powell, Executive Director

• Past DirectorsFounding Director – Prof. Larry Smarr (now in SD)Second Director – Prof. Daniel Reed (now at NC)

• A well-known and innovative organization– 280 FTEs and more than 130 students in 8 buildings, – awaiting the completion of a new building that will house

most of us.• New Director search is nearing completion


NCSA Satellite Facilities

Alliance Center for Collaboration, Education, Science, and Software

Arlington, VA

Technology, Research, Education, and Commercialization Center

Suburban Chicago


NCSA Private Sector Partners

• Previous participants– Amoco– American Airlines– Dow Chemical– Eli Lilly– FMC– Kellogg – Phillips Petroleum– Schlumberger – Shell Oil– Tribune Company– United Technologies Corp.

• Current/recent partners– Allstate Insurance – Boeing – Caterpillar– Eastman Kodak– J. P. Morgan – Motorola– Sears


International Affiliations

• NCSA’s Affiliates– COPPE, Brazil (RSN)– LNCC, Brazil (Most recent) – APAC, Australia – CCLRC, UK– KISTI, Korea– Kurchatov Institute, Russia– NCHC, Taiwan– NCSA is a member of PRAGMA – CDAC, India (in discussion)– BII, Singapore (in progress)


History and Pathways to Brazil• First visit -- NSF/U.S.- Brazilian Collaboration - August, 2002

– Talks on NCSA in multiple locations in Brazil – COPPE, FAPERJ, USP, UBrasilia etc.• Visit to NCSA by LNCC faculty – November 2002

– Bruno Schulze, Leon Sinay• ACM/IFIP International Middleware Conference 2003 – Rio, June, 2003• NSF/U.S.- Brazilian Collaboration Workshop on Advanced Materials - June, 2003• Discussions on MOA with COPPE started in July 2003

– Prof. Rochinha and Prof. Coutinho• Discussions on MOA with LNCC – August 2003

– Prof. Marco Raupp, Prof. Bruno Schulze• Visit to NCSA by COPPE faculty –August 2003

– Prof. Rochinha• Visit to NCSA by USP faculty, October 2003

– Prof. Tereza Christina Carvalho• MOA between NCSA and LNCC – December 2003• Invitation to NCSA Staff for the LNCC Workshop on Grid Computing

– Highlighted NCSA-LNCC MOA – February 2-5, 2004– Half a dozen NCSA/UIUC members participated in the workshop + our AU Affiliate

• NSF/U.S.- Brazilian Collaboration Workshop on Advanced Materials - August 2004• MOA between NCSA and COPPE – Expected to be completed in August 2004• LNCC Computational Science Workshop and Mini-symposium – next week -August

2004• Middleware workshop in conjunction with the Middleware Conference – October

2004


Recent Trends and Transitions

• New Program– Recent CISE reorganization– PACI Program sunset; New SCI Program at NSF

• NCSA’s new directions– Cyberinfrastructure in support of research, a new

imperative• New modalities

– Coopetition to cooperation– NCSA & SDSC - maintain our uniqueness and also

work together• Building Stronger Affiliations


International collaborations outlook

“The National Science Foundation should establish and lead a large-scale interagency, and internationally coordinated Advanced Cyberinfrastructure Program (ACP) to create, deploy and apply cyberinfrastructure in ways that radically empower all scientific and engineering research and allied education.”

-- NSF’s Blue Ribbon Panel /Atkin’s Report

ACP => Shared CyberInfrastructure Program


Cyberenvironments

A cyberenvironment is a subset of general CI capabilities and functionality that is designed and built to meet the needs of a particular community.

It includes use of broadly used middleware and networks as well a community specific facilities, software frameworks, networks, and people.

It is a persistent, robust, and supported capability.

Specific


Our core expertise

• Development & Deployment of Cyberinfrastructure– Computing & Grid Infrastructure– Capability and Capacity Computing– Middleware Development and Deployment

• Access to these environments for empowerment of science and user communities– Enabling breakthrough scientific discoveries– Increasing knowledge and understanding

• Community engagement– Nationally and internationally– Education, Outreach, and Training

• Building successful collaborations – To pursue all of the above


NCSA focus in Cyberinfrastructure

Stable, robust and supported cyberenvironments for scientific research and communities

• Community engagement to determine requirements• Science drivers to make sure that the requirements are

implemented correctly• R&D if/as necessary to do the development for the

requirements• Integration into the production environment• Continuing support plan of the products


Active Collaborations => Cyberinfrastructure

• Support the effective use of the extant resources by applications scientists and educators– Example: analysis of large complex datasets utilizing

on-demand and interactive resources to enable data to knowledge results

• Coordinated activities aimed at creating a national cyberinfrastructure – Partnerships and joint projects such as TeraGrid,

NCSA/SDSC collaborations, NMI, NEESgrid, …• Encourage and actively participate in

interdisciplinary collaborations


NCSA/Alliance Multiphase Strategy

• Multiple user classes– ISV software, hero calculations, data intensive analysis– distributed resource sharing, parameter studies

• Four computing approaches– shared memory multiprocessors

• 12 32-way IBM IBM p690 systems (2 TF peak)• large memory and ISV support

– TeraGrid Itanium2 clusters• 64-bit Itanium2/Madison (10.6 TF peak)• ETF partners

– IA-32 clusters (>17 TF peak)• 32-bit systems for hero calculations• dedicated sub-clusters (3 TF each)

– To be allocated for weeks or longer to specific teams– Alliance Technology Grid & Condor resource pools

• Complemented by large-scale archives– ~500 TB secondary and 2 PB tertiary storage

NCSA Control Room


NCSA Computing Environment — 32 TF

PlatinumIntel Pentium III 1 GHz IBM cluster1,024 processors1 TF peak performanceGPFS

TitanIntel Itanium 800 MHz IBM cluster320 processors1 TF peak performanceNFS

CopperIBM POWER4 p690 systems384 processors2 TF peak performanceGPFS, 24 TB

Mercury, phase 1Intel Itanium 2 1.3 GHz IBM cluster512 processors2.662 TF peak performanceGPFS, 60 TB

Mercury, phase 2Intel Itanium 2 1.5 GHz IBM cluster1,334 processors8 TF peak performanceGPFS, 170TB

TungstenIntel Xeon 3.0 GHz Dell cluster2,560 processors17.7 TF peak performanceLustre, 140 TB

#135 #111 #99

#35 #4


Extensible TeraGrid Facility

NCSA: Compute IntensiveSDSC: Data Intensive PSC: Compute Intensive

IA64

IA64 Pwr4EV68

IA32

IA32

EV7

IA64 Sun

10 TF IA-64128 large memory nodes

230 TB Disk Storage3 PB Tape Storage

GPFS and data mining

4 TF IA-64DB2, Oracle Servers500 TB Disk Storage6 PB Tape Storage1.1 TF Power4

6 TF EV6871 TB Storage

0.3 TF EV7 shared-memory150 TB Storage Server

1.25 TF IA-6496 Viz nodes

20 TB Storage

0.4 TF IA-64IA32 Datawulf80 TB Storage

Extensible Backplane NetworkLA

HubChicago

Hub

IA32

Storage Server

Disk Storage

Cluster

Shared Memory

VisualizationCluster

LEGEND

30 Gb/s

IA64

30 Gb/s

30 Gb/s30 Gb/s

30 Gb/s

Sun

Sun

ANL: VisualizationCaltech: Data collection analysis

40 Gb/s

Backplane Router

+ ETF2 sites: TACC, IU, Purdue and ORNL in FY04


Additional ETF Sites - 2004

LAHub

ChicagoHub

30 Gb/s

30 Gb/s

30 Gb/s

30 Gb/s

30 Gb/s 40 Gb/s

Cluster Agg Switch

LEGEND

Backplane Router

Linda Winkler ([email protected])

SDSC

Caltech

NCSA

PSC

ANL

20 Gb/s

AtlantaHub

20 Gb/s

IPGrid10 Gb/s

20 Gb/s

10 Gb/s

PU

IUB

IUPUI

TACC

10 Gb/s

10 Gb/s

10 Gb/s

ORNL


Combine Data Resources Across TG

Home Directory

Node Local Storage

Scratch/Staging Storage

Parallel Filesystem

Archival System Cap.

H

SDSCH

100 TB QFS64 TB GPFS

SAM-FS

2 TB NFSL

NCSAH

70 GB/node8 TB TTS

1.5 PB UniTree

1 TB NFSL

ANLH

132 GB/node IA-3216 TB PVFS

4 TB NFS

L

CaltechH

70 GB/node80 TB PVFS

1.2 PB HPSS

140 GB NFSL

PSCH

38 GB/node TCS

30 TB PFS3 PB DMF

.5 TB NFS TCSL

6 PB HPSS /

L 64 GB/node IA-64L

35 GB/node

S

S100 TB QFSS

39 TB GPFSS 24TB SLASH


Access to HPC Resources

• Allocation Process– PIs are U.S. Researchers from academic institutions– Co-PIs/Collaborators could be non-U.S. academicians– Initial small amount of resources with simple web

based requests on-line to start-out– Larger, subsequent resource allocations by proposal

submissions for peer-review– Review Boards meet frequently

• Our HPC User Community– ~2000 users per year– ~500 PIs projects– ~58% usage by MPS Division

• Math & Physical Sciences -• AST, CHE, DMR, DMS & PHY

Materials Research Projects Usage

0

500000

1000000

1500000

2000000

2500000

3000000

3500000

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

DMR


Users by Type in 2003

Grad Student38%

Faculty29%

Postdoc19%

Research Staff10%

Undergrad2%

Other2%


Cluster in a Box/OSCAR

• Community code base with strong support– Bald Guy Software, Dell, IBM, Intel, Indiana,

MSC.Software, NCSA, ORNL, Sherbrooke University• Six releases within the past year

– >29,000 downloads during this period • Recent additions

– HDF4/HDF5 I/O libraries– OSCAR database for cluster configuration– Itanium2 and Gelato consortium integration– NCSA cluster monitor package (Clumon)– NCSA VMI 2 messaging layer

• Myrinet, gigabit Ethernet and Infiniband– PVFS

• The First Annual OSCAR Symposium– May 11-14, 2003, Québec, CANADA


Grid in a Box => NMI

• Goals– middleware integration

• packaging, testing, documentation– community building

• instruments, laboratories and data

• Features and status– middleware used by many

projects– additional funding supporting

new sites

www.nsf-middleware.org


Display Wall In A Box

• Current software– Chromium and Pixel Blaster Movie Player– Argonne movie viewer– VTK geometry viewer and VNC

• Download via www.ncsa.uiuc.edu

Rear Projectors LCD Panels


Visualization of Multiple Simulations

Source: Bob Wilhelmson


NCSA’s Application Centric Efforts

• Astronomy and Astrophysics• Arts and Humanities • Biology• Engineering• Environmental Sciences• Geosciences• Integration of Education


CI Architecture

OtherservicesTeraGrid

CII

NetworkAllocation,

performancemonitoring,NWS, VMI,

GridSolve, MPI

Data-base

SMPCluster Mass Store

Network

DataMovement

GridFTP,MetaData,RFT, FileSystems

SecurityKerberos, GSI, KCA, Authorization, Credential Management, OTP, auditing, monitoring, IDS

Physical

CoreServices

Co-SchedulingCompute, Data,

Network

DataManagementSRB, HDF, Semi-

automatic MetaDatageneration, DataFusion, Sensor

Data Framework

Visualizationsystems

Application-Centric

Services

WorkFlow ServicesOGRE

Portal ServicesOGCE

User ApplicationsCactus

Data MiningSelection,

Transformations,Modeling,

Presentation,Predictive Methods,

Motif Analysis

VisualizationAutomatic FeatureExtraction, Mesa,Maya, TrajectoryViz, Professional

VisualizationServices, VTK

Collaborative ToolsChef, AG

Cyber-Environments

ResourceManagementCondor, Globus,Scheduling, On-

demand, Info Svcs,Monitoring

Tools/Libraries

J2EE, PyGlobus,CoG, Atlas,Blast, FFT...

Bio Portal...ChemistryPortal

WeatherPortal

AstroPortal


Black Hole Collision Problem

1963Hahn and Lindquist

IBM 7090One Processor

Each 0.2 MF3 Hours

1977Eppley and Smarr

CDC 7600One Processor

Each 35 MF5 Hours

1999Seidel and Suen, et al.

NCSA SGI Origin256 Processors

Each 500 MF40 Hours

300X 30,000X

1,800,000,000X

2001Seidel et al

NCSA Pentium III256 Processors

Each 1 GF500,000 Hours total

plus 500,000 hours at NERSC

~200X


Tornado Modeling – Data to Knowledge

Large complex datasets require data analysis and visualizationin the search for understanding

Wilhelmson and Cox


Tornado Simulation

Cox and Wilhelmson


Tornado Simulation – Ground Level

Cox and Wilhelmson


Data Management and Visualization Data Set

– Computations were performed on 16 processors of the IBM p690 (Regatta) supercomputer at NCSA

– ~ 16,000 time steps, 8.6 days wall clock time– Simulation spans 2.8 hours– 3D volume data available every 2 seconds around main

tornado event 6300-9600 sec (~ 55 minutes) (tornado duration ~ 40 minutes)

– Data save format HDF – compressed (save ~ 6 x space) and chunked (allows faster reading of partial volumes of data)

– Data volume ~ 650 GB in compressed and scales form (~4.0 TBytes if all data in 32 bit form)

– Animation time – 30 fps x 2 sec, therefore 1 sec animation = 1 minute in real time

Source: Cronce, Gilmore, Romine, Wilhelmson


Visualization tools and techniques

• Visualizations by the Experimental Technologies group of NCSA

• Trajectories calculated from the model dynamical fields based on original trajectory code by David Wojtowicz(DAS)

• Renderings generated with Maya software (professional rendering tool commonly used in animated movies, video games, etc…)

Source: Cronce, Gilmore, Romine, Wilhelmson


MEAD Expedition => LEAD

– Modeling Environment for Atmospheric Discovery• cyberinfrastructure for Grid-based parametric studies

– mesoscale convective systems and hurricanes• recall the Hurricane Floyd experience

– Features• WRF and ROMS coupling

– community atmospheric and ocean models• Grid workflow management• data management and visualization

– very large computed and derived data sets – high performance parallel I/O using HDF5– metadata, mining and machine learning

• model and performance analysis • education and outreach

– See www.ncsa.uiuc.edu/Expeditions/MEAD


LEAD Project Motivation

• Each year, mesoscale weather – floods, tornadoes, hail, strong winds, lightning, and winter storms – causes hundreds of deaths, routinely disrupts transportation and commerce, and results in annual economic losses > $13B.

Source: Kelvin Droegemeier


CI Underpinnings for LEAD

– On-demand– Real time– Automated/intelligent sequential tasking– Resource prediction/scheduling– Fault tolerance– Dynamic interaction– Interoperability– Grid and Web services– Personal virtual spaces

Source: Kelvin Droegemeier


Parallel Multi-Scale and Multi-Physics Simulations

micro meso

10-10 m 102 m

10-15 s 106 s

macro

macro

Source: Keshav Pingali, Cornell


Adaptive Software/Understanding Fracture

• Wide range of length and time scales• Macroscopic components used in engineering practice• Macroscopic behavior simulated using finite-element

method• Homogeneous materials at the macroscale become

heterogeneous, polycrystalline assemblies as one zooms down to mesoscales (1-10 microns)

• Structures at the mesoscale (grain boundaries, dislocation cell structures, and coalescing voids etc.) must be understood in terms of the collective behavior of large numbers of lattice defects (vacancies, dislocations, etc.)

• Atomistic modeling is required to develop effective descriptions of lattice defects, and collections of such defects, for use at the mesoscale

10-310-6 10-9 m

Source: Keshav Pingali, Cornell


Multiscale Physics

• How to do it without losing accuracy?– QMC /DFT– DFT-MD– SE-MD– FE

• How to make it parallel? (Load balancing with different methods)

< 100Å

< 1000 Å

10000 Å

Physical Scale

MacroscopicMacroscopic

MesoscaleMesoscale

Nanoscale Atomic

Nanoscale Atomic

Simulation Scale

Classical Monte Carlo

Classical Monte Carlo

Effective Mass Schrödinger Equation

Effective Mass Schrödinger Equation

Car-Parrinello Quantum Monte Carlo

Car-Parrinello Quantum Monte Carlo

Multiscale Hierarchy

MFlop

GFlop

TFlop

Integrating what is at the microscopic quantum level with the mesoscopic classical level – Great Challenge.

Lots of software and interdisciplinary work needed.

Source – David Ceperley, UIUC


Questions?

http://www.ncsa.uiuc.edu

[email protected]

a glimpse of ncsa’s role in support of research

Documents