Projects using Cactus

Gabrielle Allengallen@cct.lsu.edu

Cactus RetreatBaton Rouge, April 2004

Cactus Projects @ AEI/LSUCactus Projects @ AEI/LSU

User Support Applications Research and Development

Support for more numerical models (Saturday)

Large scale computing Frameworks Visualization Data models and formats Grid computing

Community DevelopmentCommunity Development Adaptive Mesh Refinement

PAGH (Wash U) Carpet (Albert Einstein Institute)

Application Performance Modeling University of Chicago

Data Formats and Visualization Lawrence Berkeley Lab Konrad Zuse Zentrum Albert Einstein Institute

Optimization and Performance NCSA, Intel, Cray, Lawrence Berkeley Lab, Absoft

User GUIs Wash U

InteroperabilityInteroperability

Pending proposal to NSF ITR program “Hypercode: Interoperable Infrastructure Initiative”

Addressing interoperability for general computational infrastructures, Cactus, Chombo, Paramesh and CCA-based frameworks

Focused around applications: Numerical Relativity Computational Astrophysics Coastal Modeling Climate Modeling Cosmology Computational Fluid Dynamics

InteroperabilityInteroperability

InteroperabilityInteroperability

Develop common mechanisms and abstractions to enable different simulation codes to use modules and data interchangeably

Partners: LSU LBL (John Shalf, Phil Colella, Julian Borill) U. Maryland (Kevin Olsen, Joan Centrella) U. Indiana (Denis Gannon) NCSA (Greg Daues)

InteroperabilityInteroperability

Main developments for Cactus: Incorporate other AMR drivers into Cactus

– Chombo, Paramesh Incorporate other elliptic solvers into Cactus Develop a community toolkit for CFD Add new features to Cactus

– Adaptivity– Dynamic component loading

Develop common data model– New visualization tools available

VisualizationVisualization

Ongoing visualization projects at AEI, LSU, LBL with Cactus

Pending DST-NSF proposal with the computer science department and C-DAC to build visualization infrastructure to allow data to be analyzed and visualized on the fly Additions to Cactus I/O infrastructure Thorns for visualization Web-based visualization tools

Large Scale ComputingLarge Scale Computing

NSF Software Technologies for High End Computing (being written)

Incorporate fault tolerant MPI in Cactus driver layer University of Tennessee

Develop HTTPD thorn into an interactive, real time parallel debugger

Detect and exploit memory and network connection hierarchy from processor cache, through node layout on clusters of SMPs, to cluster interconnections on the Grid

Performance monitoring and adaption using e.g. PAPI library

Grid ComputingGrid Computing

… infrastructure enabling the integrated, collaborative use of high-end computers,

networks, databases, scientific instruments owned and managed by multiple

organizations …

… applications often involve large amounts of data and/or computing,

secure resource sharing across organizational boundaries, not

easily handled by today’s Internet and Web infrastructures …

“resource sharing and coordinated problem solving in dynamic, multi-institutional virtual organizations”

Remote Monitoring/Steering: Remote Monitoring/Steering: HTTPD Thorn which allows

simulation to any to act as its own web server

Connect to simulation from any browser anywhere … collaborate

Monitor run: parameters, basic visualization, ...

Change steerable parameters Running example at www.

CactusCode.org Wireless remote viz, monitoring

and steering

User Portal (ASC and GridSphere)User Portal (ASC and GridSphere)

Collaboration focal point for a virtual organization

Interact, share data Start jobs on remote

resources Move/browse files Track and monitor

announced jobs Access to new Grid

technologies www.ascportal.org www.gridsphere.org

Portal Server

SMS Server

Mail Server

“TestBed”

Running

Appli cations

Notification (Announce)Notification (Announce)

OpenDX, Amira, …

HDF5

GridFTP VFD

Stream VFD

Visualization Tools

GridFTP

Remote Data Server

IOStreamedHDF5

Simulation

Hyperslabbing, Downsampling

Remote Data and Visualization (GriKSL)Remote Data and Visualization (GriKSL)

Dynamic Adaptive Distributed ComputationDynamic Adaptive Distributed Computation

SDSC IBM SP1024 procs5x12x17 =1020

NCSA Origin Array256+128+1285x12x(4+2+2) =480

OC-12 line

(But only 2.5MB/sec)

GigE:100MB/sec

17

12

5

4 2

12

5

2

These experiments: Einstein Equations (but could be

any Cactus application)Achieved:

First runs: 15% scaling With new techniques: 70-85%

scaling, ~ 250GF

“Gordon Bell Prize” (Supercomputing 2001,

Denver)

Dynamic Adaptation: Number of ghostzones, compression, …

New Grid ScenariosNew Grid Scenarios Intelligent Parameter Surveys, Monte Carlo Dynamic Migration: faster/cheaper/bigger

machine Multiple Universe: create clone to investigate

steered parameter Automatic Component Loading (Needs of process

change) Automatic Convergence Testing Look Ahead Spawn Independent/Asynchronous Tasks Routine Profiling: best machine/queue,

parameters Dynamic Load Balancing: inhomogeneous loads,

multiple grids

GridLabGridLab

Cactus experiments with grid computing on the E-Grid Cactus Worm: thorns which allowed simulations

to migrate themselves from machine to machine Spawning: sending (asynchronous) calculations in

analysis thorns to different machines We wrote the GridLab proposal to be able to do

these, and other scenarios, in a better way

GridLab ProjectGridLab Project

http://www.gridlab.org EU Funded ($5M) by 5th Framework

January 2002-December 2004): Many partners in Europe and US

PSNC (Poland), AEI & ZIB (Germany), VU (Netherlands), MASARYK (Czech), SZTAKI (Hungary), ISUFI (Italy), Cardiff (UK), NTUA (Greece), Chicago, ISI & Wisconsin (US), Sun/Compaq

LSU now a collaborating partner

GridLabGridLab

Developing an easy-to-use, flexible, generic and modular Grid Application Toolkit (GAT), enabling applications to make innovative use of global computing resources

Focused on two principles: co-development of infrastructure with real

applications and user communities (Badly needed in grid computing!!)

dynamic use of grids, with self-aware simulations adapting to their changing environment.

GridLabGridLab

12 Work Packages covering:Grid PortalsMobile Users

Different Grid ServicesApplications(Development) Test Bed

Grid Application Toolkit (GAT)

Grid Application ToolkitGrid Application Toolkit

Need a layer between applications and grid infrastructure:

Higher level than existing grid APIs, hide complexity, abstract grid functionality through application oriented APIs

Insulate against rapid evolution of grid infrastructure and state of grid deployment

Choose between different grid infrastructures Make it possible for application developers to use

and develop for the grid independent of the state of deployment of the grid infrastructure

Monitoring

Resource Management

InformationSecurity

DataManagement

GLOBUS

ApplicationManager

Logging

NotificationMigration

Profiling

SOAP WSDL Corba OGSA Other

Other GridInfrastructure?

Application

“Is there a better resource I could be using?”

Application

“Is there a better resource I could be using?”

GAT_FindResource( )

The Grid

The Same Application … The Same Application …

Application

GAT

Application

GAT

Application

GAT

Laptop The GridSuper Computer

No network! Firewall issues!

GAT: Grid Application ToolkitGAT: Grid Application Toolkit

Standard API and Toolkit for developing portable Grid applications independently of the underlying Grid infrastructure and available services

Implements the GAT-API Used by applications (different languages)

GAT Adaptors Connect to capabilities/services

GAT Engine Provides the function bindings for the GAT-API

http://www.gridlab.org/software/GAT

GAT ArchitectureGAT Architecture

Cactus/GAT IntegrationCactus/GAT Integration

GATLibrary

Cactus Flesh

Thorn

CGATThorn

Thorn

Thorn

Thorn

Thorn

Physics and Computational Infrastructure

Modules

Cactus GAT wrappers Additional

functionalityBuild system

GridLab Service

GridLab Service

Grid ScenarioThorn

Grid ScenarioThorn

TFM

TFM TFM TFM TFM

Task Farming on the GridTask Farming on the Grid

TFM implementedin Cactus

GAT used for starting remote TFMs

Designed for the Grid

Tasks can be anything

Grid-Black HolesGrid-Black Holes

Task farm small Cactus black hole simulations across testbed

Parameter survey: black hole corotation parameter

Results steer a large production black hole simulation

Physicist has new idea !

S1 S2

P1

P2

S1S2

P2P1

SBrill Wave

Dynamic Grid ComputingDynamic Grid Computing

Found a horizon,try out excision

Look forhorizon

Calculate/OutputGrav. Waves

Calculate/OutputInvariants

Find bestresources

Free CPUs!!

NCSA

SDSC

RZG

LRZ

Archive data

SDSC

Add more resources

Clone job with steered

parameter

Queue time over, find new machine

Archive to LIGOpublic database