cornell theory center aug 15 2000 cctk the cactus computational toolkit werner benger...
TRANSCRIPT
![Page 1: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/1.jpg)
Cornell Theory Center Aug 15 2000
CCTKThe Cactus Computational
ToolkitWerner Benger
Max-PIanck-Institut für Gravitationsphysik(Albert-Einstein-Institute at Golm/Potsdam – AEI)
andKonrad-Zuse-Center for
Information Technology Berlin (ZIB)[email protected]
![Page 2: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/2.jpg)
Introduction
Cactus: Original Motivation Numerical Relativity
Ongoing Research
What is the Cactus Computational Toolkit? Cactus Design - Flesh and Thorns
What does Cactus provide?
The Cactus Framework How to use Cactus
Interfacing Cactus
Future Developments Metacomputing for Numerical Relativity
Plans...
![Page 3: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/3.jpg)
Cactus - Original Motivation: Numerical Relativity
Kepler: stable ellipsesEinstein: no stable orbits because of gravitational radiation(GR required for extreme gravity)
Astronomy with gravitational wave detectors (taken into operation by this year 2000!)
The Two-Body problem in General Relativity is still unsolved.
![Page 4: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/4.jpg)
Axial symmetric Collisions of Black Holes (NCSA 1993-96)
Pre-Cactus (H-Code, G-Code), Cactus 1.0, 2.x, Cactus 3.xUS - NSF Grand Challenge Projects
Requirement for Collaborative Code
A distorted single black hole
Two Black Holes
![Page 5: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/5.jpg)
Focus of 3D Numerical Evolutions during the last couple years
Black Holes (prime source for GW) – from Misner BH collisions to grazing BH collisions with initial spin and momentum (Brandt-Bruegmann initial data)
Gravitational Waves (Evolution of Brill Waves, collapse of pure GW, investigation of critical amplitude, i.e. when do black holes form)
Neutron Stars (NASA Neutron Star Grand Challenge, GR hydrodynamics, neutron stars colliding to black holes,...)
![Page 6: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/6.jpg)
Visualization of 3D data with ZIB's Amira
Efforts for Remote Visualization, Remote Monitoring, Remote Steering German Gigabit Project (TIKSL), KDI Portal
![Page 7: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/7.jpg)
German Gigabit Project: TIKSL
Remote Visualization
Remote Steering
Online Monitoring
Globus Services
HDF5 Interface
Data Grid Access (DPSS)
![Page 8: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/8.jpg)
What is the Cactus Computational Toolkit?
Portable Application Framework
"Flesh" provides registration and scheduling facilities
Memory management ("gridarrays","gridfunctions")
"Thorns" are exchangeable code segments
Runtime activation/deactivation of thorns
I/O layers (parallel I/O)
Parameter handling
Exchangeable Parallelization layers (MPI, Globus-MPI, Shared Memory,...)
![Page 9: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/9.jpg)
Data Types (Portability)
Cactus data types to provide portability across platforms
CCTK_REAL CCTK_REAL4, CCTK_REAL8, CCTK_REAL16
CCTK_INT CCTK_INT2, CCTK_INT4, CCTK_INT8
CCTK_CHAR
CCTK_COMPLEX CCTK_COMPLEX8, CCTK_COMPLEX16, CCTK_COMPLEX32
![Page 10: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/10.jpg)
Cactus Flesh
Make System Organizes builds as configurations which hold everything needed to build
with a particular set of options on a particular architecture.
API Functions which must be there for thorns to operate.
Scheduler Sophisticated scheduler which calls thorn-provided functions as and when
needed.
CCL Configuration language which tells the flesh all it needs to know about the
thorns.
![Page 11: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/11.jpg)
Data Structures (Memory Management)
Grid Arrays A multidimensional and arbitrarily sized array distributed among
processors
Grid Functions A field distributed on the multidimensional computational grid (a Grid
Array sized to the grid)
– Every point in a grid may hold a different value “f(x,y,z)”
Grid Scalars Values common to all the grid points
Parameters Values/Keywords that affect the behavior of the code (initialization,
evolution, output, etc..)
– parameter checking, steerable parameters
![Page 12: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/12.jpg)
Cactus Thorns
Flesh (core) written in C Thorns (modules) grouped in packages written in F77, F90, C, C++ Thorn-Flesh interface fixed in 3 files written in CCL (Cactus
Configuration Language): interface.ccl: Grid Functions, Arrays, Scalars (integer, real, logical, complex) param.ccl: Parameters and their allowed values schedule.ccl: Entry point of routines, dynamic memory and communication
allocations
Object oriented features for thorns (public, private, protected variables, implementations, inheritance) for clearer interfaces
Compilation: PERL parses the CCL files and creates the flesh-thorn interface code at
compile time Particularly important for the FORTRAN-C interface. FORTRAN arg. lists
must be known at compile time, but depend on the thorn list
![Page 13: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/13.jpg)
Interface
The concept: contract with the rest of the code Now it is only for the data structures: variables and parameters adding routines and arguments
Private The variables that you want the flesh to allocate/communicate but
no other thorn to see.
Public The variables that you want everybody to see (that means that
everybody can modify them too!) Inheritance
Protected Variables that you want only your friends to see!
![Page 14: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/14.jpg)
Implementation
Why Two or more thorns that provide the same functionality but different
internal implementation– Interchangeable pieces that allow easy comparison and evolution in
the development process
– They are compiled together and only one is activated at runtime
How If all the other thorns need to see the same contract, then thorns
implementing a certain functionality must– Have the same public and protected variables
– The same concept applies to parameters and scheduling
Example Wildly different evolution approaches for the same equations, so all
the analysis and initial data thorns remain the same.
![Page 15: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/15.jpg)
Scheduling
Thorns schedule when their routines should be executed
Basic evolution skeleton idea standard scheduling points INITIAL, EVOL, ANALYSIS fine control: run this routine BEFORE/AFTER that routine
Extend/customize with scheduling groups Add my routine to this group of routines Run the group WHILE some condition is met
Future redesign The scheduler is really a runtime selector of the computation flow. Much more power can be added to this concept
![Page 16: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/16.jpg)
Parallelizing an Application Thorn
CCTK_SyncGroup– synchronize ghostzones for a group of grid variables
– just add Synchronization to Scheduler configuration file as well
CCTK_Reduce– call any registered reduction operator, e.g. maximum value over the
grid
CCTK_Interpolate– call any registered interpolation operator
CCTK_MyProc– unique processor number within the computation
CCTK_nProcs– total number of processors
CCTK_Barrier– waits for all processors to reach this point
![Page 17: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/17.jpg)
PUGH
The standard parallel driver supplied with Cactus is supplied by thorn PUGH
Driver thorn: Sets up grid variables, handles processor decomposition, deals with processor communications
1,2,3D Grid Arrays and Grid Functions (beta6)
Uses MPI
Custom processor decomposition
Otherwise decomposes in z, then y, then x directions
![Page 18: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/18.jpg)
How to use Cactus
[Optional: Develop thorns, according to some rules e.g. specify variables through interface.ccl
Specify calling sequence of the thorn subroutines for given problem and algorithm (schedule.ccl)]
Specify which thorns are desired for simulation (e.g. Einstein equations + special method 1 +HRSC hydro+wave finder + AMR + live visualization module + remote steering tool…)
Specified code is then created, with only those modules, those variables, those I/O routines, this MPI layer, that AMR system,…, needed (minimal binary)
Subroutine calling lists generated automatically
Automatically created for desired computer architecture
Run it…(local, remote, on the Grid using Globus environment)
![Page 19: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/19.jpg)
How does Cactus help ?
Collaborative working
Different people are experts on different parts of the physical problem
Each person can write a thorn which solves their part, e.g. the metric evolution, hydrodynamics, apparent horizon finding, ...
Thorns are encapsulated, and different thorns with the same functionality are interchangeable
Parallelism
Cactus provides a parallel layer
Layer is independent of underlying machine architecture
Researchers don't need to think deeply about parallelism Choice of parallel library layers (Native MPI, MPICH, MPICH-G(2), LAM,
WMPI, PACX, HPVM, MPIPro)
![Page 20: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/20.jpg)
...
IO Cactus provides optimized IO layers for the various machines
Possible to output very large datasets in a short time
Parallel I/O (Parallel HDF5, Panda, John Shalf's FlexIO - various interfaces to MPI-I/O)
Checkpointing A lot of runs take more time than queing systems allow
Cactus provides mechanisms to dump out the entire state of the simulation and then to read it in again either on the same machine or another
Platform independence
Cactus provides a platform independent environment
Various "strange compilation" issues on different machines are already handled by the CCTK environment
![Page 21: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/21.jpg)
CACTUS on NT
The NT Supercluster was used in three demos and two of the HPC challenges at SC ’98 , held in Orlando, Florida,
November 9-13, 1998.http://www.ncsa.uiuc.edu/General/CC/ntcluster/sc98/
![Page 22: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/22.jpg)
Who Uses Cactus
Numerical Relativity AEI
WashU
NCSA
Penn State
UIB
Southampton
PRL
...
Computer Science Panda IO Project (UIUC)
Globus (Argonne)
Cluster evaluation
– Roadrunner (UNM)
– NT cluster (NCSA)
– ...
Autopilot (UIUC)
Gigabit Project (DFN)
![Page 23: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/23.jpg)
Horizons (A Metacomputing Application)
Singularity at center hidden by event horizon Surface through which nothing in the interior can escape
Event horizon only really detectable if we have the whole spacetime
Not possible while running the simulation
Apparent horizon always within event horizon Many methods to detect this
If a horizon is found Definitely have a black hole Can compute Gaussian Curvature to inspect oscillations and correlation to
emitted gravitational waves
![Page 24: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/24.jpg)
Distributed Computation of AH
SpaceTime Evolution
gmn t, K mn t
Apparent Horizon Computation
AH(t=9.0)
AH(t=11.0)
AH(t=16.0)
RZG CRAY T3E, 512 Nodes(Garching/Munich)
ZIB T3E, 16 Nodes
AEI Origin 2000, 16 Nodes
Cornell NT Cluster, 16 Nodes
GlobusServices
All the required technique (TIKSL NetHDF5, Globus MDS Queries, ...) is already there!
![Page 25: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/25.jpg)
Cactus communication layer Parallel driver thorn (e.g. PUGH) currently provides both variable
management and communication … abstract send and receives etc
Abstract communication from driver thorn easily implement different parallel paradigms shared memory, threads, Corba, OpenMP, PVM, ...
Compact groups (different layout in memory for improved Cache performance)
Unstructured Meshes/Finite Elements/Spectral Methods Unstructured Multigrid Solver Convergence/Multiple Coordinate Patches Capability browsing mechanism Command line interface … connect directly to Cactus,
scheduling GUIs, Documentation, GUIs, Documentation ….
Coming up (Cactus 4.2)
![Page 26: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/26.jpg)
What physical systems are there now ?
Initial Data Schwarzschild
Misner
Brandt-Bruegmann puncture data
Brill waves
Teukolsky waves
Distorted Brill wave and black hole
TOV
Colliding neutron stars
Orbiting neutron stars
Boson star
Dust
![Page 27: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/27.jpg)
...
Analysis Apparent horizon finders
fast flow
minimisation
Wave extraction
Riemann Invariants
Newman-Penrose quantities
Constraint evaluation
Geodesic tracking
![Page 28: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/28.jpg)
Overview
Introduction Cactus - Original Motivation: Numerical Relativity Axial symmetric Collisions of Black Holes
(NCSA 1993-96) Focus of 3D Numerical Evolutions
during the last couple years Visualization of 3D data with ZIB's Amira German Gigabit Project: TIKSL
![Page 29: Cornell Theory Center Aug 15 2000 CCTK The Cactus Computational Toolkit Werner Benger Max-PIanck-Institut für Gravitationsphysik (Albert-Einstein-Institute](https://reader033.vdocument.in/reader033/viewer/2022051619/56649e445503460f94b38131/html5/thumbnails/29.jpg)
Overview, II
What is the Cactus Computational Toolkit? Data Types (Portability) Cactus Flesh Data Structures (Memory Management) Cactus Thorns Interface Implementation Scheduling Parallelizing an Application Thorn PUGH How to use Cactus How does Cactus help ? CACTUS on NT Who Uses Cactus Horizons (A Metacomputing Application) Distributed Computation of AH