an advanced simulation & computing (asc) academic strategic alliances program (asap) center at...

An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center

at The University of Chicago

The Center for Astrophysical Thermonuclear Flashes

Transitioning from FLASH2 to FLASH3

Katie Antypas

June 4, 2006

The ASC/Alliances Center for Astrophysical Thermonuclear FlashesThe University of Chicago

Overview

FLASH3 Enhancements

Transitioning to FLASH3

Where to get help


FLASH3 Motivation

Make FLASH capable of being a community developed and supported code by cleaning up code architecture and defining clear unit interfaces

Build on positive aspects of FLASH2 Make FLASH code everlasting even if FLASH Center

isn’t. First opportunity to focus on design rather than

capabilities


FLASH3 Goals

Configurable Flexibility to easily configure the code for new applications A traditional strong point of FLASH

Extensible Users can add functionality quickly

Verifiable Users should be able to test their applications easily The FLASH test suite was release as a stand alone application in

FLASH3 Well Documented

A necessity for any community code that hopes to gain broad acceptance FLASH3 include:

Extensive online API documentation Improved Users Guide Examples of transitions between FLASH2 and FLASH3


FLASH3 Release

Includes all infrastructure units (Grid, IO, Driver, etc) Includes basic physics units (Hydro, gamma EOS) Extensive Web-based documentation

User’s guide Reference Tips Howto’s Developer’s guide Public Unit API documentation Runtime parameters documentation

Test Suite, including use of unit test framework


Future Planned Releases

Closely spaced to make new functionality available to users as soon as possible

Alpha release September 2006 Particles (Lagrangian and active) Nuclear burning Eos Helmholtz Simple gravity MHD

Beta release December 2006 3.0 release April 2007

Full FLASH2 functionality with new FLASH3 capabilities


FLASH3 Enhancements : Interchangeable Grids

Paramesh

Uniform Grid

One block per processor No AMR related overhead

Patch Based Mesh(planned)

Adaptive mesh with variable block size

Block Structured Fixed sized blocks Specified at compile time Not more than one level jump

at fine course boundaries


FLASH3 Enhancements : API Interface

Each unit has a well defined API (in CS terms each unit has a set of public routines which other units can call)

Examples for Grid unit: Grid_fillGuardCells Grid_conserveFluxes Grid_updateRefinement Grid_moveParticles Grid_get/putBlkData Grid_getNumProcs

Each routine is fully documented with argument and algorithm descriptions!!


Robodocs API Grid_conserveFluxes


FLASH3 Enhancements : Decentralized Data

In FLASH2 centralized database was the key way to access all data Positives: easy for developers to get data they needed quickly

and easily Negatives: not clear who owned the data, when it was last

modified or if it was safe to modify data In FLASH3 in lieu of a centralized database each unit

contains a FORTRAN module to hold data specific to that unit Still have accessor functions so developers can get data

easily Now more clear who owns the data Much of the data stored in the FLASH2 database is located in

the Grid_data module in FLASH3 or in the Flash.h file, (more later)

Holds myPE, numProcs, access to grid variables, coordinates etc.


Example Flash.h File

#define DENS_VAR 1#define EINT_VAR 2#define ENER_VAR 3#define GAMC_VAR 4#define GAME_VAR 5#define PRES_VAR 6#define TEMP_VAR 7#define VELX_VAR 8#define VELY_VAR 9#define VELZ_VAR 10#define MFRAC_SPEC 11

#define NPROP_VARS 10#define NSPECIES 1#define NMASS_SCALARS 0#define NUNK_VARS (NPROP_VARS + NSPECIES + NMASS_SCALARS)

#define E_FLUX 1#define EINT_FLUX 2#define P_FLUX 3#define RHO_FLUX 4#define U_FLUX 5#define UT_FLUX 6#define UTT_FLUX 7

#define NPROP_FLUX 7#define NSPECIES_FLUX 1#define NMASS_SCALARS_FLUX 0#define NFLUXES (NPROP_FLUX + NSPECIES_FLUX + NMASS_SCALARS_FLUX)

#define NDIM 2#define MAXBLOCKS 1000#define TEMP_SCRATCH_GRID_VAR 1#define NSCRATCH_GRID_VARS 1

The Flash.h file is written by the setup script for your specific application

Declares variable and flux indices in multidimensional datastructures

Defines, number of dimensions, maximum blocks on a proc and much more ...

variable indices

flux indicies


Example Putting Data onto Grid

FLASH2

FLASH3

idens = dbaseKeyNumber(‘dens’)call dbasePutData(idens, ....)

use index directlycall Grid_putBlkData(DENS_VAR, ....)


Grid Directory Structure

GridAPI level

•Stub File ie. No implementation•All routines called from other Units belong in the API

GridMain

UG paramesh

PM2 PM3

Grid_fillGuardCellsGrid_getCellVolumeGrid_conserveFluxesGrid_getBlkNeighborsGrid_getGlobalNumBlksand many more...

Grid_getGlobalNumBlks

Implementation levels

•All other levels hold the implementation for the API routines

•Both the Uniform Grid and Paramesh have the same implementation for Grid_getGlobalNumBlks

•The Uniform Grid has no implementation for Grid_conserveFluxes and so when that routine is called from the Hydro Unit, it just returns the empty stub

Grid_getBlkNeighborsGrid_conserveFluxes


FLASH3 Unit Structure

Distinguish between public and private routines Public Routines

Must have stub implementation Naming convention, start with “Unit_verbWhatUnitDoes” ie

“Grid_putBlkData” Private Routines

No stubs unless necessary for inter-subunit communication Naming convention, small case abrev. for unit ie “gr_createDataTypes”

Each unit has a ‘UnitName’_data module to hold unit scope data Each unit has a ‘UnitName’_init routine to initialize data stored in

‘UnitName’_data module All initializations that were in “if firstcall = .true.” blocks, now belong in

the “UnitName_init” routine which is called once at initialization time

In FLASH2, the user was free to organize a code unit however he or she wanted with very few constraints. In FLASH3 we

implemented a number of ‘strong conventions’ (rules) for easier code maintenance, development and usability.


Moving a FLASH2 module to FLASH3

Create a ‘UnitName’_data FORTRAN module to store data needed from other units in the code.

Create storage in ‘UnitName’_data for all necessary variables storing runtime parameters or physical constants.

Eliminate the “first call” constructs, move them to the UnitName_init.

Use wraper functions to fill up the remaining variables in Unit_data.


Impact on user

We’ve taken steps to assure the transition isn’t too painful Lots of documentation of Flash code support website If you are having any problems please email us. We can

help. Most kernels are the same as in FLASH2 A user, who has modified source files in setup:

should be able to read Unit API documentation to clearly determine how to make the code compatible with FLASH3 or

Look at the new source file implementation for how to start on converting to FLASH3

If you’re in the process of developing a deep functional module for FLASH2 that you want to contribute, talk to us now


FLASH3 Enhancements: Unit Testing

FLASH3 has a unit testing framework and comes with several unit tests for the Grid, RuntimeParmeters, EOS, Multispecies and PhysicalConstants

Users can easily add in new unit tests Attempt to test units independently, however some units can not

be meaningfully tested alone Most unit tests require the Grid unit Multigamma Eos, requires Eos and Multispecies

Setup examples: ./setup unitTest/Eos -auto ./setup unitTest/Grid -auto

Simple Driver can be written to test unit

In any large complex code it is often difficult pin point where coding problems exist. Unit testing allows small sections of code to be tested independently, targeting the debugging process and

giving the user more confidence in the complete application.


FLASH3 Enhancements: The New Test Suite

Flash test suite runs a variety of problems to validate the code on a daily basis

Essential for immediately identifying bugs unintentionally introduced into the code

New test suite was released with FLASH3 so users can monitor their own research problems and performance test parameters easily modified through GUI handles unit tests, compilation tests, comparison tests automatically uploads test suite data to benchmarks database

Green light indicates all runs were successful

Date of run

Platform

Floating statistics box gives immediate overview of results

Red light indicates 1 or more tests failed


Editing of “test.info” file done right in browser

“test.info” file holds all necessary information for running a FLASH

problem

Expandable graphic allows user to easily modify all problems in test suite

Control Panel for modification of

“test.info” file tree

Info File Manager


Test Suite Performance History

The new test suite adds the ability to upload selected testlog files to the database to track daily code performance.

History of 2D and 3D Sedov Problems Details of 2D History


FLASH3 Enhancements : Enhanced Setup Script

Seamless library management Ability to retain object files from one setup to the next Suggests units a user might have neglected Users can setup shortcuts

More about setup script options in next talk


FLASH3 Framework

How do you get the Framework out of FLASH 3? Include the source tree up to the top level of each

Unit. Stub implementations Config files

Include source/Simulation/SimulationMain/Default directory

Include the setup tool Include the DriverUnit tests could be included, but have no meaning

with stubs


… which brings us to

Discussion and Questions

an advanced simulation & computing (asc) academic strategic alliances program (asap) center at...

Documents