High performance and data-intensive computation at Caltech/CACR

Dan Meiron

Acting Director, CACR

Associate Provost for Information and Information Technology

Role of simulation

• Extension of conventional theory-experiment basis• Complex systems play a role in almost all areas• Computational power continues to grow exponentially• Caltech’s role

– Strong foundation in experimental and theoretical science– Long track record in developing and exploiting computation– Significant fraction of research relies on large scale

distributed computing• Large scale scientific computation• Scalable algorithms and data structures• Data assimilation

Computational Science and Engineering

• Related endeavors in simulation and the intellectual underpinnings comprise the new area of Computational Science and Engineering (CSE)

• Inherently interdisciplinary area• Challenges

– Optimal utilization of massively parallel systems– Design of scalable algorithms– Interplay between algorithms and architecture– Software development

• Core areas– Applied mathematics– Computer science– Strongly driven by applications

CSE Applications

• Representation from all Caltech divisions– Biology

• Gene sequencing• Protein design

– Physics• Simulation and detection of gravitational sources• Observational astronomy and cosmology

– EAS• Continuum mechanics• Materials

– CCE• First principles analysis of matter• Simulation of reactions

– GPS• Climate• Seismic simulation

– HSS• Economic modeling

• Current CSE search focused on core aspects of CSE

CACR is a team of dedicated individuals committed to solving

computational problems on the

frontiers of science and engineering

through collaboration with the worldwide

research community.

By following an applications-driven

approach to computational science

and engineering,

By providing an intellectual

environment that cultivates

multidisciplinary collaborations,

By harnessing technologies to

create innovative large-scale computing

environments

And by conducting multidisciplinary

research on these leading-edge

computing facilities.

Parallel computing systems

Storage systemsNetworking

Visualization

CACR Computing Resources

CACR is building collaborations to extend the

frontiers of science and engineering

through advanced computing research.

Building the TeraGrid

Caltech is a partner in this

effort to build and deploy the

world’s largest, fastest, most

comprehensive, distributed

infrastructure for open scientific

research.

Featured CACR Collaborations and Projects

Data Intensive Distributed

Computing for Physics

Caltech & CACR play major roles in the

planning, research, development and

management of several well-funded US and

international grid-based projects for physics.

Virtual Astronomy

The Virtual Sky project brings together

astronomical image resources from many

agencies and wavelengths into the

same projection. Virtual Sky is part of the NSF-

funded National Virtual Observatory, a five-year

effort to federate astronomical data

resources.

www.virtualsky.org

Montage

• deliver science-grade custom mosaics on demand,

• portable, compute-intensive service

• applied in the first instance to the 2MASS, SDSS and DPOSS image data sets.

• Will run operationally on the emerging Terascale Facility,

Caltech DTF System 32

Cisco 6509Myrinet 2000

32

0.5 TF/0.384 TB Memory32 Compute Nodes

8

243 x 10 GbE

Login Node

8

24

8 Datawulf NodesPVFS, 21.6 TB

24 Datawulf NodesData Cache, 64.8 TB

 

Datawulf Node (2 x IA-32)

Gigabit MyrinetEthernet 2000

ATARAID

ATARAID

ATARAID

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

150 GB

45

2xSunT3

2xSunT3

2xSunT3

2xSunT3

9840/9940Tape

Drives

9

HPSSNode

1

HPSSNode

2

HPSSNode

3

HPSSNode

4

DTFNetwork

ExistingCACR

Servers

4

1

Gigabit Enet 10 Gigabit Enet Myrinet Fibre Channel

24

Cisco 6509

ExtremeBlack Diamond

(existing)

24

Juniper M160(existing)

OC-48 POSNTON

OC-12 ATMCampus

Collaborators

Dan MeironActing Director

Jim PoolExecutive Director

Michael AivazisDirector, Research

Sarah Emery-Bunn

Chip Chapman

Roy Williams

Mark Bartelt

Tom Gottschalk

Thomas Sterling

Julian Bunn

Dave Breen

Heather Young

CACR organization 1

CACR – organization 2

Jim PoolExecutive Director

Special ProjectsJim Pool

Chip ChapmanSarah Emery-Bunn

Computing ResourcesJ. Pool, Interim Manager      

M. Bartelt, TeraGrid Site Leader C. Chapman, Manager, Facilities

P. Angelino  S. Brunett* A. Carasik 

M. Feldmann* J. Lindheim

J. Morris J. Patton

AdministrationHeather Young, Manager

Monty AdamsTracy ShefferSusan PowellWendy Ward

Michael AivazisDirector, Research

Software Development and Frameworks

Michael Aivazis, Group LeaderSharon Brunett

Fehmi CirakJulian Cummings

Thoutireddy Pururav

Grid ApplicationsRoy Williams, Group leader

Julian BunnMike FeldmannJames Patton

VisualizationD. Breen, Group Leader

S. Lombeyda J. McCorquodale

K. Museth

Advanced SystemsT. Sterling, Group Leader

J. BrockmanM. Brodowicz T. Gottschalk E. Upchurch

H. Zima

CACR – organization 3

Caltech ASCI center - CACR collaboration

• Virtual shock physics facility – explore full dynamic response of

target materials to wide range of loadings

• compressive• tensional• shear

– loading generated by• high velocity impact - strong

shock waves• detonation of high explosive

– facilitate full three dimensional simulation

– validate these computations against experiment

• Associated faculty– Michael Ortiz (ME, GALCIT)– Joe Shepherd (GALCIT)– Paul Dimotakis (GALCIT)– Dale Pullin (GALCIT)– Peter Schroder (CS)– Jason Hickey (CS)– G. Ravichandran (GALCIT)– Ares Rosakis (GALCIT)– Bill Goddard (Chemistry)– Raul Radovitzky (MIT)– Alberto Cuitino (Rutgers)– Manish Parashar (Rutgers)– Ron Cohen (Carnegie)– Emily Carter (UCLA)

The Virtual Test Facility

Multiscale modeling (solid mechanics)

hours

minutes

seconds

microsec

nanosec

picosec

femtosec

time

distance Å nm micron mm cm meters

Lattice defects

Dislocation dynamics

Multiphysics,Multicomponent

SystemsSubgrain

structures,polycrystals

Solver integration – M. Aivazis

custom application driver

properties

geometry

Python bindings

pyadlib.so

MPDb VTFApplication

Controller Application

Mesher Solver StrengthModel

Pyre

materials

meshing

adv. features viz. support

fem checkpoints

VTFApplication

Integrated applications using Python

MPDbMaterialModel

arm3d

pyarm3d

GrACE

pygrace

ACIS

pyacis

ARM3d

solid engine

pyadlib

Adlib

options

GUI

script

InterfaceManager

Geometry

Solver

Sensor/Probe

Monitor

Controller

Application

MPI

pympi

Dynamic response of solids

Taylor anvil testprobes plasticity

Intermediate length scales

• Modeling at intermediate length scales:– Subgrain structures

• Effective behavior• Scaling, size effect

Subgrain structures in shocked Ta (Meyers et al, 1995)

Polycrystalline Ta (Murr et al., 1997)

Shock propagation in polycrystalline Ta

Slip activity

Detonation diffraction around a cube

LLNL Blue on 128 processors. 3 AMR Levels.

VTF3d Scaling – LLNL (Blue Pacific)

VTF3d HE-TA Scaling 48^3 grid/cpu, 45K element solid mesh

01020304050

0 200 400 600 800 1000

fluid cpus

se

c/t

ime

ste

p

fluid solver

fluid boundary

VTF Vision

Experiment/validationIntegrated scalable simulations

Integrated Simulation Environment Advanced modeling

CACR and DANSE

• DANSE fits well with Caltech/CACR initiatives– Software integration

frameworks

– Large scale distributed facilities

– Strong science base in materials research