hydra: a new paradigm for astrophysical modeling ...hydra: a new paradigm for astrophysical...

20
HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael A. Nowak, Dan Dewey, John E. Davis, and Claude R. Canizares Abstract Regardless of the source or mission, the analysis of astrophysical data invariably relies on the interplay between predictive physical models, a detailed understanding of the observing instru- ments, and the discriminating comparison of predictions and actual observations. With HYDRA, we are developing a new platform which will provide scientists a more flexible and extensible way of constructing models for astrophysical sources that include geometric information, physical emis- sion and absorption mechanisms, transport processes, and projection effects. We will also provide an interface through which users may link existing astrophysical models to HYDRA. Similarly, modules describing the performance of the instrumentation, be it ground-based telescope or orbit- ing satellite, will be definable in a mission-independent way to allow realizations of these models in the form of simulated observations. By combining source and instrumentation models, HYDRA can serve as a tool for observational planning, instrument design, and calibration. Simultaneously, HYDRA will provide an analysis environment that allows source models to be compared to existing observations and iteratively adjusted. As part of its design, the HYDRA system will also include advanced visualization capabilities that will provide users with additional diagnostic ability as well as a potentially powerful educational tool.

Upload: others

Post on 03-May-2020

9 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

HYDRA: A New Paradigm for Astrophysical Modeling,Simulation, and Analysis

Michael W. Wise, Michael S. Noble, John C. Houck, Michael A. Nowak,Dan Dewey, John E. Davis, and Claude R. Canizares

Abstract

Regardless of the source or mission, the analysis of astrophysical data invariably relies on theinterplay between predictive physical models, a detailed understanding of the observing instru-ments, and the discriminating comparison of predictions and actual observations. With HYDRA,we are developing a new platform which will provide scientists a more flexible and extensible wayof constructing models for astrophysical sources that include geometric information, physical emis-sion and absorption mechanisms, transport processes, and projection effects. We will also providean interface through which users may link existing astrophysical models to HYDRA. Similarly,modules describing the performance of the instrumentation, be it ground-based telescope or orbit-ing satellite, will be definable in a mission-independent way to allow realizations of these modelsin the form of simulated observations. By combining source and instrumentation models, HYDRAcan serve as a tool for observational planning, instrument design, and calibration. Simultaneously,HYDRA will provide an analysis environment that allows source models to be compared to existingobservations and iteratively adjusted. As part of its design, the HYDRA system will also includeadvanced visualization capabilities that will provide users with additional diagnostic ability as wellas a potentially powerful educational tool.

Page 2: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael
Page 3: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

Contents

1 Motivation 1

2 Current Analysis Systems 1

3 Research Goals 2

4 Advanced Source Models 3

4.1 Beyond Spectral Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

4.2 Current HYDRA Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

4.3 External Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4.4 Photon Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4.5 Distributed Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

5 Multi-Mission Simulations 6

5.1 Simulation Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

5.2 Hardware Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

6 Visualization 7

7 Flexible Fitting 9

8 Implementation 11

8.1 Modular Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

8.2 Programmability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

8.3 Polyglot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

8.4 Large Scale Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

9 Management Plan 14

9.1 Development Milestones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

9.2 Investigator Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

9.3 Equipment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

References 16

Page 4: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael
Page 5: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

1 Motivation

Future astronomy missions will require technicaladvances on many fronts to achieve their scien-tific objectives. The need for increased collect-ing area, spatial resolution, and spectral resolv-ing power will drive development in a number ofareas including optics and detector technology.Similarly, the increasing complexity of the datafrom these missions will require correspondingadvances in the capabilities of the software toolswe use to analyze and interpret that data. Ifwe are to maximize the potential science returnfrom our mission investments, we will need flex-ible analysis tools which can accommodate andexploit this complexity.

Current missions such as Chandra and XMM,with their mixture of spatial and spectral infor-mation, are already posing data analysis chal-lenges using current analysis paradigms (e.g.,Peterson et al., 2002). Missions such asConstellation-X and Astro-E2 will only increasethis level of complexity, with observations thatmix components of spatial, spectral, and tem-poral analyses. See for example, Young andReynolds (2000) which describes the “rever-beration mapping” analysis of relativisticallybroadened Fe Kα lines - a prime goal of theConstellation-X mission. Inherent in these pro-posed analyses is an “event based” approach thatof necessity will allow users to combine complexsource models with detector simulations that ac-curately reproduce all of the observatories prop-erties, spatial, spectral, and temporal. Concur-rent with these NASA led efforts to develop thenext generation of models and observatories, ef-forts must be made to develop analysis tools withthe capabilities to match.

With HYDRA, we are exploring a newparadigm for astronomical analysis systems thatwill address these needs. By exploiting a va-riety of modern computing techniques, we aredeveloping a system which provides the capabil-ity to define sophisticated source models, highlyaccurate instrument simulations, and customiz-able analysis methods, all within the framework

of a flexible and extensible platform which canadapt to the needs of current missions as wellas those to come. The many complex facets ofthis endeavor recall the mythical, many-headedHydra. Our goal is to develop an analysis sys-tem which, by design, takes into account thefact that modern astronomical analysis is mul-tidimensional, multiwavelength, and ultimatelymulti-mission.

Such a system supports NASA’s objectives de-tailed in “The New Age of Exploration: Direc-tion for 2005 and Beyond”, specifically Objec-tive 51 and Objective 132. Studying the forma-tion and evolution of systems from black holes toclusters of galaxies, tracing the chemical historyof the universe as it leads to life, and exploringthe behavior of matter in extreme environmentsare all goals which require a new level of sophis-ticated modeling and analysis taking advantageof available advances in computer technology.

Also, by extending the general tool kit ofthe astronomer to routinely include 3D model-ing and visualization, the HYDRA system canhelp bridge the gap between ongoing scientificresearch and the public. The interactive 3D as-pects of the HYDRA system in particular canprovide an engaging educational tool for stu-dents allowing them to model and explore realand imagined astrophysical systems.

2 Current Analysis Systems

Before continuing, it is worth placing our HY-DRA conception in the context of current analy-sis systems. An assortment of software tools areavailable to astronomers, with AIPS/AIPS++,CIAO, MIDAS, EXSAS, FTOOLS, IDL, IRAF,PROS, STSDAS, and XANADU among themore prevalent. The majority of these, however,are tailored to either a specific type of analysis

1Explore the universe to understand its origin, struc-ture, evolution, and destiny.

2Use NASA missions and other activities to inspire andmotivate the Nation’s students and teachers, to engageand educate the public, and to advance the scientific andtechnological capabilities of the nation.

1

Page 6: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

or an individual type of data. Arguably, muchof their functionality is devoted more to datapreparation and reduction, accounting for the in-strumental characteristics of a particular missionor observatory, than actual analysis. The capa-bilities they do provide are often specific to oneclass of analysis, e.g. image or spectral, but notboth. HYDRA is intended to complement ex-isting packages by providing an advanced fittingand analysis environment much in the manner ofXSPEC, ISIS3, and Sherpa, the analysis enginesof their respective packages.

Unlike these tools, which presume spectraldata of a form describable by binned counts withassociated response and effective area files, HY-DRA will further abstract these concepts so thatdata events can be compared to model eventsobtained from a realistic, full observatory sim-ulation. Thus, HYDRA will not be limited tocomparing a model to a spectrum, but will alsobe able to compare models to images, temporalinformation, combinations of all three, or evencustomized aspects of datasets defined by the in-dividual user (see §7).

Another aspect of their common origins as X-ray spectral analysis tools is that all three pack-ages include source models which consider onlyspectral emission. There is currently no simpleprovision for modeling the spatial structure ortemporal properties of astrophysical sources. Aswe discuss in §4, providing this capability in ageneral and flexible way is one of the main goalsof the HYDRA project. This same flexibility willallow users to link in their own potentially moresophisticated models with little overhead.

In terms of flexibility, the ISIS (Houck andDenicola, 2000) analysis package is one of themost easily extensible astrophysical analysis sys-tems currently available, with other systems typ-ically requiring significant programming skill toincorporate new models, new statistical meth-ods, etc. The recently released XSPEC v.12 hasfinally adopted this model, employed by ISIS forseveral years, wherein new functionality can beintroduced without recompiling the system, pro-

3Developed by John Houck at the MIT Kavli Institute.

vided that the new code has been written to con-form to the new, more generalized XSPEC inter-faces. We note, however, that XSPEC v.12 stillpresumes spectral data in the form of 1D arraysof binned counts with associated response andeffective area files, and that its control language(Tcl) is highly unsuited for numerical work.

Finally, we note that none of the currentlypopular systems incorporate advanced visualiza-tion capabilities. With HYDRA, we plan to pro-vide an analysis platform with new capabilitiesin all of the areas outlined above.

3 Research Goals

With the discussion of the previous section inmind, we can outline a number of specific goals:

• Provide a general and flexible means ofdefining more realistic, multidimensionalmodels for astrophysical sources.

• Provide a system which allows the behaviorof multiple missions to be simulated easilyand with appropriate fidelity.

• Provide a flexible environment which canadapt to the specific needs of a wide rangeof scientific analyses.

• Incorporate advanced statistical meth-ods for the analysis of complex, multi-dimensional datasets.

• Incorporate modern visualization tech-niques to refine both source model construc-tion and analysis decisions.

• Incorporate mechanisms which fostergreater use of high performance computingmethods within desktop analysis.

The remainder of the proposal will concentrateon our specific plans for the design and imple-mentation of the HYDRA system.

2

Page 7: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

4 Advanced Source Models

Giving the user the ability to easily constructmore realistic models of astrophysical sourcesand use these models directly in the analysis ofobservational data is one of the primary goalsof the HYDRA project. Source models specifythe incident photon spectrum as a function ofposition in the field of view and, optionally, asa function of time. In the most general case,the incident flux at a given position is computedby integrating contributions from various modelcomponents arranged along the line of sight, ac-counting for geometry and all relevant absorp-tion and emission processes.

4.1 Beyond Spectral Models

Current spectral analysis packages such asXSPEC, SHERPA and ISIS provide a simple,intuitive means to construct complex spectralmodels using a library of simpler components.For example, in ISIS, given source spectrummodels A and B, a source which contains con-tributions from both can be modeled using theexpression of the form A(1) + B(1). In this ex-pression, the numerical indices identify a spe-cific instance of each spectral shape. Absorptionalong the line of sight may be modeled by anenergy-dependent multiplicative factor since theeffect is to reduce the source flux. Combiningthese components, one might define a complexsource model using a simple syntax of the form:

model = W(1)*( (A(1)+B(1))

+ W(2)*(A(2)+B(2)) )

This expression specifies two multicomponentsources, each having a different amount of ab-sorption along the line of sight. In general, mul-tiplicative components provide an approximatetreatment of foreground absorption and all addi-tive components are assumed to be optically thinso that they add linearly and may be summedwithout restrictions.

While this approach is convenient for combin-ing spectral components, no package currently

provides support for including models with spa-tial structure.

4.2 Current HYDRA Prototype

We have developed prototype software that com-putes spectra incident on a telescope from an ar-bitrary 3D distribution of emitting and absorb-ing material in the field of view. This softwareis currently implemented as a generalized sourcemodel for ISIS. Prototyping this model withinISIS has a number of immediate advantages in-cluding the ability to use any of the emissiv-ity and opacity functions currently available forXSPEC. In addition, we can take advantage ofthe native capabilities in ISIS for applying in-strumental responses and minimizing fit statis-tics when comparing models to data. Conse-quently, we can begin the process of testing andrefining this prototype immediately without hav-ing to wait for development of the other majorcomponents of HYDRA.

Beyond accounting for absorption along theline of sight, we have not attempted to treatself-consistently the microphysics of the couplingbetween radiation and matter. We assume thatthe emissivity and opacity are specified every-where in the source volume, usually through pa-rameterized functions suitable for model fitting.Models which require detailed radiation transfereffects are beyond the scope of our initial efforts.

For generality and computational convenience,the source volume is subdivided into a Cartesiangrid, and assigned a coordinate system orientedwith lines of sight (LOS) along the Z axis asshown in Figure 1. This scheme simplifies thecomputation of integrals along the line of sight.To compute incident spectra for a different view-point, one could either consider lines of sight atan angle to the Z axis or rotate the emissivityand opacity distributions within the fixed coordi-nate grid to place the desired line of sight alongthe Z axis. In our current prototype, we havechosen to implement the latter approach. Thischoice makes it easier to distribute the compu-tation over multiple CPUS.

3

Page 8: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

∆Z : cell width

LOS

, η , χ , ∆Z )jI = I ( I j−1 j j

η

contribution of j−th cell given by

j : opacity of j−th cell

: emissivity of j−th cell

Figure 1: Schematic representation of the generalized 3D spatial-spectral model implemented in the currentHYDRA prototype and discussed in the text. Integration through the grid along a given line of sight yieldsthe observed, projected intensity as a function of position and energy.

The prototype provides a S-Lang API to sup-port defining source models as 3D distributionsof energy-dependent emissivity and opacity. Thesource volume is implemented as a 3D array ofstructures, one structure for each spatial zoneor cell. Each structure contains the energy-dependent emissivities and opacities for the cor-responding cell. When the source model consistsof multiple components, some cells may containa list of several emissivity and opacity contri-butions. If the source model has a high degreeof symmetry (e.g. spherical symmetry), one canexploit the symmetry to reduce the memory stor-age requirements.

In our prototype, the emissivity and opac-ity structure fields are implemented as arraysof pointers. When no model component con-tributes emission or absorption in a given cell,the corresponding structure fields in that cell aresimply null pointers. In cells containing contri-butions from one or more components, the ar-rays in that cell contain pointers to the relevantemissivity and opacity curves. This scheme ismemory efficient because there is no duplicationof emissivity and opacity curves. For maximumefficiency, the code to manage this 3D array ofstructures is implemented in C; higher level op-erations are implemented in S-Lang.

While general 3D models require definingemissivity and opacity curves for each cell in

the source volume, models with a high degree ofsymmetry can be defined by providing much lessinformation. To simplify the definition of simple3D models, we will provide a number of commondistributions. The existing prototype providestriaxial ellipsoids, thereby including spheres andellipsoids as obvious special cases. To define aspherical model, the user must provide emissiv-ity and opacity curves as a function of radius.A common approach might be to define physicalvariables as a function of radius (e.g. a power-law radial temperature dependence). At eachradius, one would then compute the emissivitiesand opacities using spectral models like thoseprovided by XSPEC. To define a 3D model, theuser would select a set of emissivity models (e.g.from among the XSPEC models) and would thenprovide the radial dependence of every model pa-rameter (or accept the default parameter value).

4.3 External Models

To support analysis of source models which aremuch more complex than those outlined above,we plan to provide a reasonably general soft-ware interface that can support a wide varietyof sources. In general terms, such an interfacemust provide a stream of incident photons withspecified direction, energy, and arrival time andat an appropriate rate at each point in the field

4

Page 9: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

of view. We plan to adopt an interface similarto the one defined to support user-defined sourcemodels in MARX4.

For example, a hydrodynamic simulation ofan astrophysical source could define the physicalconditions throughout a source volume (e.g. thetemperature and density distribution for X-rayemitting gas in a cluster of galaxies). HDF5 for-matted output files from such a simulation couldbe loaded into a S-Lang script using our recentlydeveloped HDF5 module. Spectra incident upona telescope could be computed by line-of-sightintegration over volume emissivities and opaci-ties as described in §4.2. Given these incidentspectra, a MARX-style Monte-Carlo simulationcould be driven by generating a stream of inci-dent photons with the appropriate distributions.Utilizing the generalized software interface de-scribed here, this model could be connected tothe rest of the HYDRA system, visualized us-ing its 3D rendering capabilities, folded througha simulation of various observatories, and castin the form of simulated data for comparison toactual observations.

4.4 Photon Generation

To generate the incident photon stream for agiven model, it is sufficient to provide an inter-face which takes a vector representing a line ofsight and returns the corresponding photon dis-tribution. In one approach, related to the imple-mentations used by XSPEC, SHERPA and ISIS,the model photon distribution may be foldedthrough the instrument response to generate apredicted data set. This “forward-folding” ap-proach is a mechanism for “solving” the integralequation discussed in §5 which relates the inci-dent source flux to detected quantities.

Alternatively, one can drive a Monte-Carlosimulation by randomly drawing incident pho-tons from the specified distributions which de-scribe the spectral and spatial properties of a

4MARX is a Monte-Carlo simulator that accuratelymodels the behavior of the Chandra X-ray Observatory,developed at MIT by members of this collaboration.

Figure 2: A map of the amount of ICM heating inthe center of the Perseus cluster. It was calculatedfrom a deep Chandra observation using a customizedscript in ISIS (written in S-Lang) which automatesthe process of extracting and fitting spectra. The im-age consists of 104 individual fits and was calculatedin only several hours using the PVM module on anetwork of 25 workstations (Wise and Houck, 2005).

given source. With this approach, the instru-ment response can be treated using a mission-independent hardware abstraction layer (see thediscussion of ARDLIB in §5). We plan to ex-plore these alternatives and, given that each ap-proach has advantages in certain situations, mayultimately elect to support both techniques. Tomaximize run-time performance, the subroutineinterface for computing source models will sup-port caching pre-computed quantities. Beforemodel computations begin, a function call willperform any required initialization.

4.5 Distributed Processing

Although our aim is to create an analysis systemthat can run quickly and efficiently on a typicalworkstation, we plan to support larger scale sim-ulations that may require more computationalresources. The cheapest approach for large scalesimulations is to distribute the work load overa small local network of computers such as may

5

Page 10: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

be found in most University astronomy depart-ments. We have extensive experience with dis-tributed processing of X-ray data (see Figure 2)using ISIS and PVM (Geist et al., 1994).

In our work on large scale simulations, we willfocus on computations which are inherently par-allel, such as Monte-Carlo simulations, which areideally suited for distributed processing.

5 Multi-Mission Simulations

5.1 Simulation Modules

In an era of proliferating astronomical dataarchives, the ability to work with data from mul-tiple missions or observatories is a crucial re-quirement for any analysis system. Scientifically,data analysis is becoming an increasingly multi-wavelength endeavor and our tools must reflectthis need. Although some of the data analysispackages in current use attempt to meet thismulti-mission goal, they often do so at the ex-pense of fidelity. The detailed and potentiallycomplex performance of instruments and obser-vatories are typically included using highly sim-plified representations. Consequently, subtle orcomplicated effects present in actual observa-tional data are often neglected. When these sim-plified instrument models are then coupled withalready overly simplified source models, the re-sults often bear only a qualitative resemblanceto the actual data they are meant to represent.These simplifications place limitations on boththe type and accuracy of scientific analyses.

As part of its design, HYDRA will include theability to couple the advanced source models dis-cussed in the previous section with simulationsof the observational performance of a given ob-servatory. The resulting simulated observationscan be used for a variety of purposes includ-ing observational planning, instrument perfor-mance studies, and most importantly, scientificanalysis. Building upon our development of theChandra X-ray simulator MARX, we have devel-oped a mission-independent protocol for specify-ing instrument models by which the same source

model may be folded through a simulation ofmany different observatories. In principle, mul-tiple simulations for a given observatory whichreflect different levels of fidelity can be deployedusing the same interface depending on the typeof analysis desired, i.e., one might trade fidelityfor computational speed.

By following this protocol, simulation modulesfor new missions or observatories can be added asneeded. We currently have a high fidelity, simu-lation module in place for the Chandra X-ray ob-servatory based on MARX. Our immediate plansare to expand to include both the XMM-Newtonobservatory and Astro-E2 missions as well as aprototype for the proposed Constellation-X mis-sion. Modules for other observatories and mis-sions will be added as HYDRA evolves. Al-though our initial focus will be on existing andfuture X-ray missions, we will investigate and as-sess the feasibility of including other wavebandssuch as radio, optical, and infrared as well. Ourgoal will be to design interfaces of sufficient flex-ibility to allow for the potential differences inphoton behavior and characterization possible inthese diverse domains.

5.2 Hardware Abstraction

Instrument simulation in HYDRA may beviewed as a means of providing a generalizedsolution to the integral equations (Davis, 2001)that relate the flux from an arbitrary distri-bution of source photons to observable quanti-ties such as detector count rates. These equa-tions contain functions that characterize the var-ious aspects of the instrumental response tosource photons. Examples of such responsesinclude the telescope effective-area, the point-spread function, CCD quantum efficiencies, andgrating efficiencies. Because the instrumenta-tion varies from mission to mission, the re-sponse functions are inherently mission-specific.ARDLIB5 is a software library that representsthese mission-dependent functions through amission-independent application programming

5Developed by John Davis at the MIT Kavli Institute.

6

Page 11: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

interface (API) (Davis, 2000). This abstractionis made possible through the object-oriented de-sign of the library. ARDLIB has been adoptedby the Chandra data analysis system (CIAO)and is used with great success by the programsthat create Chandra effective areas and expo-sure maps. Similarly, ARDLIB will also play afundamental role in the mission-independent re-alization of HYDRA.

Ultimately, the user would need to know verylittle about ARDLIB to use HYDRA effectively.In fact, one of the most important conceptsof object-oriented design is that of data en-capsulation. ARDLIB uses this concept effec-tively to hide the mission-dependent as well asimplementation-specific details about how in-strumental responses are computed. Rather,the ARDLIB public interface makes referenceto objects that have mission-independent mean-ings. For example, instead of having a func-tion that computes Chandra’s HRMA effectivearea, which requires knowledge that Chandrahas four mirror shells, and so forth, ARDLIBsimply deals with the instantiation of an abstracteffective area object, which in itself has a well-defined meaning independent of any specific in-strument. Of course the effective area computedfrom the abstract object would correspond tothe appropriate instrument-specific quantity. Inother words, if one is using HYDRA to analyzeXMM data, then that object will represent theXMM effective area.

The encapsulation of mission-specific detailsinto abstract objects also means that HYDRAwill automatically support a new mission oncesupport for that mission has been added toARDLIB. That is, not a single line of HYDRA’scode would have to be modified to gain such sup-port. One immediate consequence of using suchan object-oriented library is that software main-tenance of HYDRA will be vastly simplified. An-other side-effect of this approach is that the HY-DRA programmers themselves will need to havevery little knowledge about the various missionssupported by HYDRA.

Finally, it is worth noting that ARDLIB is

a thread-safe library. As mentioned previously,for maximum performance HYDRA will allowthe user to exploit multiple CPUs using PVMto distribute a computation over a network ofmachines. Alternatively, for a single machinewith multiple processors, one may want to usea threaded version of HYDRA to distribute theload across the CPUs. As multi-cpu machinesare quite common, the fact that ARDLIB isthread safe could provide HYDRA with signif-icant performance enhancements.

6 Visualization

The ability to visualize both the input sourcemodels and the resulting simulated observationsrepresents an important new capability offeredby the HYDRA system. Datasets from modernmissions are increasingly complex and often bet-ter suited to analysis using more sophisticatedrepresentations than traditional 2D plots andimages. To begin to address this need, we havealready incorporated within the HYDRA proto-type two forms of visual data exploration whichwe believe are not standard components of anypublic astronomical package.

The first of these is VWhere (Noble, 2004b),which supports the interactive creation of com-plex, multi-dimensional filters via the applica-tion of regions to plots of dataset slices (Fig-ure 3). This approach can instantaneously revealdata correlations that can be difficult and time-consuming to discover with traditional commandline tools, while requiring exactly zero knowledgeof package-specific filtering syntax. Based uponthe powerful where function built-in to S-Lang,VWhere exceeds traditional tools by a wide mar-gin in terms of its fluid ease-of use, runtime per-formance, and scope of analytical power (e.g. itis impossible to extend compiled, command linefiltering tools at runtime in order to utilize a newroutine contained in, say, the GNU scientific li-brary to filter an event list).

As we discuss in the next section, HYDRA willinclude the ability to utilize user defined data fil-tering “on the fly” during analysis itself, rather

7

Page 12: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

Figure 3: The interactive filtering tool VWhere, part of the current HYDRA prototype, in action. The leftpanel shows a polygonal filter being applied to events in a color–intensity diagram. The right panel showsthese selected events (indicated in red) in the corresponding light curve.

than only within the preceding data reductionphase. These filters may take the form of morefamiliar aspects of the data, such as images orspectra, but may also include more esoteric prop-erties (see for example Figure 5 in §7). The abil-ity to visualize, and cut, datasets in potentiallycomplex ways will be valuable to users who re-quire special purpose filters uniquely suited totheir particular analysis. VWhere is a first steptoward giving users this ability in HYDRA.

A second way in which HYDRA will exceedthe capabilities of current packages is by incorpo-rating volume visualization as a standard com-ponent. As part of its education and public out-reach mission, NASA has identified the devel-opment of unique and compelling new teachingtools as one of its strategic goals. By giving sci-entists the capacity to visualize the actual, work-ing models they use in their analyses, HYDRAhas the potential to be a powerful teaching toolfor communicating with teachers and students

Towards this end, we have created the proto-type volview guilet (see Figure 4), based uponthe Volpack rendering library (Lacroute and

Levoy, 1994). With this module we are able tovisualize both input source models as well as theresulting simulated observations, an importantcapability not provided by any current package.The construction of source models which includespatial components (such as those described in§4) is likely to represent a more subtle process,potentially requiring interactive fine tuning; vi-sualizing model components during these itera-tions provides a useful “sanity check,” and canlead to new qualitative insights.

Volpack was selected for our initial volumetricwork because it is very small, has no external de-pendencies, requires no special hardware, and isvery fast. Its shear-warp factorization algorithmis generally recognized as the fastest renderingtechnique available (Meißner et al., 2000), a cru-cial factor for interactive analysis. The adoptionof Volpack, however, does not preclude eventu-ally incorporating packages with a wider range offunctionality, such as OpenGL or Vtk. Further-more, the HYDRA user remains free to utilizethe S-Lang interface to Python and import ex-isting visualization tools such as PIL or MayaVI.

8

Page 13: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

Figure 4: Volumetric rendering of a simple 3D emis-sivity model using the volview module. The emissiv-ity model was computed using the HYDRA prototype3D model discussed in §4.

7 Flexible Fitting

An analysis environment must be able to de-fine flexible models, compare them to observeddata, and adjust them in order to determinethe desired underlying physical quantities. Cur-rent systems implementing these iterative com-parisons tend to be geared toward a specific typeof analysis, e.g. spectral, and are not easily con-figured for other types of comparisons. Conse-quently, users must resort to using multiple sys-tems to perform different types of analysis on thesame dataset. This division reflects more currentsoftware design, rather than the scientific needsof the user.

HYDRA will be designed to allow users tohave nearly complete control over the analysisprocess. This is an ambitious task, but onewe believe to be achievable by taking advantageof improvements in computational power (e.g.,faster CPUs linked via PVM) and advances instatistical methods. The latter will be a specificresearch focus in the early stages of our proposedprogram. Modern advances in statistics and in-formation theory (e.g., Bayesian techniques) willbe explored with two goals in mind: giving usersthe ability to gauge and visualize the ‘informa-

tion content’ of their data, and giving users theability to choose which aspects of the data willdrive the fitting process.

Traditional “one dimensional” fitting algo-rithms (e.g., XSPEC, Arnaud, 1996) typicallyrely upon the user binning the data to a min-imum counts per bin so that subsequently amodel can be applied and a χ2 derived. Regard-less of the resolution of the detector, the ‘effec-tive resolution’ is determined by the number ofbins with adequate counts. An alternative ap-proach is to instead use Bayesian techniques (fora review geared towards modern astronomicalapplications, see Connors, 2003) wherein modellikelihoods are calculated based upon the actualstatistical properties of the data (which are of-ten governed by Poisson, rather than Gaussian,statistics). This approach, however, can be com-putationally expensive, especially for users whowish to fit multi-dimensional (e.g., spatial andspectral) data simultaneously.

One approach to reducing the complexity ofthe required calculations is to first allow the userto gauge the information content of their data, sothat the user then can decide upon the degree offidelity that can or should be incorporated intotheir desired models. A current example of thisapproach using Chandra data are ‘wavelet’ basedmethods for detecting image structure (Freemanet al., 2003), which then allows a user to con-centrate modeling efforts only on those regionswith sufficient structure. The modular approachof HYDRA, whereby different instrument mod-els can be inserted into the analysis path, willallow such procedures to be easily adapted toother instruments.

Bayesian methods allow more generalized ap-proaches toward finding structure in multi-dimensional data. Scargle (2003) describesBayesian methods of sub-dividing data intomulti-dimensional ‘blocks’ with statistically sig-nificant structure. These methods have beenrefined to be both computationally less expen-sive and more robust (Scargle et al., 2004). Wehave created S-Lang implementations6, for one

6Code, examples and descriptions can be found at

9

Page 14: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

dimensional data, that have been successfullyapplied to finding structure in both time seriesdata (e.g., flares) and within gratings data (e.g.,absorption and emission lines)7. The latter ex-ample is illustrative of the modular approachthat HYDRA will take. Event data are takenfrom the observations, model data are createdwithin an analysis engine (e.g., ISIS), and thenthe two are passed to and compared within anindependent statistics module (i.e., the S-Langsubroutines). This approach will allow us toquickly and easily adopt new statistical tech-niques. We even have explored and identifiedways to incorporate statistical modules writtenin other scripting languages, such as Python(e.g., Loredo, 2001). Again, it is the indepen-dence and modularity of data, instrument model,and statistical analysis, that makes this possible.

The second, and perhaps more difficult, com-ponent of the statistics problem is to identifymethods for fitting multi-dimensional models tomulti-dimensional data. This issue will be amajor focus of research for us as the HYDRAproject progresses. We believe that one key tothis challenge will be to allow the user as muchcontrol as possible over what aspects of the datadrive fits to each of the parameters of the model.An illustrative, although somewhat specializedand “hardwired”, example of this can be found instudies of X-ray emission from clusters of galax-ies (Peterson et al., 2001; Peterson et al., 2002;Peterson et al., 2003).

The XMM-Newton Reflection Gratings Spec-trometer (XMM-RGS) is designed to pro-duce high-resolution spectra (λ/∆λ ≈ 500 at1 keV=12.4 A) of point sources (e.g., sourcessmaller than the ≈ 15′′ spatial resolution of theXMM mirrors). X-ray clusters, on the otherhand, with core radii of O(1′), are spatially ex-tended relative to the XMM resolution. Thedispersed X-ray image, however, still has an ef-

http://space.mit.edu/CXC/analysis/SITAR.7The ‘Bayesian block’ approach to structure identifi-

cation can readily be adapted to multi-dimensional datasets (Scargle 2005, priv. comm.), and has some overlapwith the previously cited wavelet techniques. We plan onadapting our own implementation to 2D and 3D.

fective resolution of λ/∆λ ≈ 100: a factor of≈ 7 better than native CCD resolution (Petersonet al., 2001, 2003). There is thus additional infor-mation content in the data which current anal-ysis systems are unable to exploit. The meth-ods outlined by Peterson et al. (2002) illustratenecessary components of the proposed HYDRAsystem, and suggest other potentially desirablefeatures.

A necessary aspect of this work is the ability totake a complex model (e.g., spatially-dependentand extended input spectra) and fold it througha detector model that reproduces the propertiesof observed data with high fidelity. Whatever“analysis decisions” and statistics one choosescan then be applied to the observed and modeleddata alike, which allows meaningful comparisonsto be made. This ability is the core philosophy ofthe HYDRA system, as in marked contrast to,for example, XSPEC, where many analysis de-cisions (e.g., binning of the data, temporal cutson the data, etc.) must be determined before thedata is input to the analysis engine.

The second approach adopted by Petersonet al. (2002), which highlights at least a desiredfeature of the HYDRA system, is to allow theuser to determine which aspects of the modelfits are to be associated with different propertiesof the data. For example, Peterson et al. (2003)use the ‘cross dispersion’ profile of the data (i.e.,detected counts per pixel in directions perpen-dicular to the gratings dispersion direction) tofit the cluster ‘core radius’, while along the dis-persion direction counts are collapsed into a sim-ple one-dimensional description of count rate vs.wavelength. The latter is fit in a simple one-dimensional χ2 sense, albeit with a two spatialdimension (although azimuthally symmetric) in-put model. The problem is then reduced from a2-D to a 1+1-D comparison.

This is clearly not the desired end-point formulti-dimensional fit comparisons. Future mis-sions will require true multi-dimensional fitting,wherein the dimensions might represent differ-ent data types, such as spectral and temporaldata inter-mixed (see, for example, Figure 5).

10

Page 15: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

Figure 5: A complex spectral/temporal model (left) that one might expect to fit to Constellation-X data(simulated on the right). The model represents a discrete flare occurring above an accretion disk surroundinga super-massive Schwarzschild black hole. The flare is reprocessed as a fluorescent Fe Kα line which isrelativistically smeared and time delayed. The model on the left shows the amplitude response of thesmeared line as a function of both photon energy and time. The right shows a realistic simulation of aConstellation-X observation. The model and data are inherently 2-D and must be fit in a 2-D fashion, withthe fit relying on an accurate simulation of instrument events. (Figures from Young and Reynolds 2000.)

It does, however, highlight the necessity for themodel and observed data to be treated in analo-gous manners, and for the user to have flexibilityin defining fit statistics independent from the in-put models and the detector model. By adoptinga modular approach, we will be able to exploredifferent methodologies for defining fit statistics.

8 Implementation

Computer-aided desktop analysis remains thecore activity of the observational scientist, yetwe note that while there exists a wealth of tech-niques for advanced visualization and high per-formance computation, a significant gap remainsbetween what can be achieved in these areasand what is regularly utilized by astronomerson the desktop. Limiting factors of conceptualcomplexity and implementation buy-in conspireto inhibit the addition of advanced computingmethods to the standard analytical toolbox, withthe result that typical astronomers take onlyfractional advantage of such (Noble, 2003).

8.1 Modular Architecture

Drawing from our experience, as both user anddeveloper, with major astronomical softwarepackages, we note a tendency in extant systemswhereby algorithms or libraries are internallycoupled in ways which dramatically constraintheir use elsewhere. Such systems, while open inthe superficial sense that sourcecode is publiclyavailable, are in a deeper sense closed by theircomplex web of internal dependencies. Poten-tial users confront the prospect of sifting throughtens or hundreds of megabytes of software ordata simply to obtain a small library or algo-rithm; this effectively prevents the use of novelportions of a system apart from its whole. Con-versely, tightly coupled architectures betray adistinct evolutionary disadvantage, in that theyalso tend to inhibit, rather than promote, the in-corporation of software developed externally tothe system.

HYDRA will thus be constructed not as aclosed, monolithic entity, but rather as a trulyopen and extensible system, woven from indi-vidual components that will, to the fullest ex-tent possible, remain orthogonal in function. We

11

Page 16: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

expect to write some of these ourselves, buthope that others result from the work of out-side groups. As a result, substantial portions ofHYDRA will not be developed from scratch, butrather realized by connecting existing softwarein new ways or by evolving current tools.

In form and function HYDRA will drawlargely from the heritage of ISIS. Originally writ-ten to analyze Chandra spectra, ISIS has evolvedinto a highly capable platform for general anal-ysis, is already in wide use, and for over 4years has served a crucial role in crystallizingmany of the ideas espoused here. In contrastwith much of the analysis software used by as-tronomers today, modularity and extensibilityhave, since inception, been a bedrock elementof ISIS, and are the fundamental design princi-ples of the HYDRA architecture. For example,the HYDRA main() routine is essentially a thinwrapper around the S-Lang interpreter, used pri-marily to establish callbacks to both static- anddynamically-loaded modules, as well as gatheruser input. (Figure 6).

Taking modularity to this extreme, wherecomponents provide the bulk of HYDRA’s ana-lytical capabilities, provides for an uncommonlyclean and flexible implementation. It also sup-ports incremental delivery of benefit to the com-munity, in the sense that new components mayalso be distributed independently, well beforea new version of HYDRA proper need be re-leased. To wit, most of the modules describedhere are already publicly available, and may thusbe used independently of HYDRA, even withincompletely foreign analysis systems 8.

8.2 Programmability

We note the resurgence of interest in interpretedlanguages over the past decade. The notion ofusing a scriptable environment as glue to bindindividual components — each developed in thelanguage most suited to the task — into a co-

8As demonstrated by the use of VWhere (Noble,2004b) from Python, at GLAST (Hydra web site, 2005)or the Perl Inline::SLang module.

herent whole has become enormously popular(Ousterhout, 1998), due in large part to its in-nate flexibility and promise for increasing bothdeveloper and end-user productivity. Designedfrom the outset with this in mind, two of themost compelling features of ISIS are its pro-grammability and extensibility, both endowed bythe embedded S-Lang scripting language.

S-Lang, which forms the core of numerousopen source projects, and is bundled with everyLinux distribution, provides most of the usualbenefits one expects from a scripting languageHowever, it is S-Lang’s innate mathematical ca-pability which makes it truly suited for our work.

For example, S-Lang natively supports com-plex numbers, as well as highly optimized, andtransparently vectorized, operators and func-tions. This fosters the use of concise analyticexpressions such as

c = sin(a) + b / 10.0

without regard to whether a and b arescalars, vectors, or 4-dimensional arrays, andwith performance and capability on par withcompiled code and commercial software. Thefact that these features are built in to S-Langdistinguish it from more widely known languageslike Perl, Python, or Tcl, which natively lackhigh-performance multidimensional numericalcapability. As indicated by Bagley (2003),native S-Lang operators can outperform theselanguages by two orders of magnitude or moreon representative calculations (e.g. matrixmultiplication), while our references (and in-house benchmarks) further indicate that nativeS-Lang arithmetic operations can perform up totwice as fast as hand-crafted C/C++ extensionscreated specifically to optimize numerical workin Python.

The idea here is not that S-Lang is the bestlanguage for science or general purpose scripting.Our point, rather, is simply that S-Lang is wellsuited to our problem domain, and in our expe-rience well worth the price of admission. It bearsa strong resemblance to IDL, and our colleagues

12

Page 17: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

AtomicDB

Emissivitymodule

ASCIII/O

module

Fittingmodule

Modelingmodule

PGplotmodule

PGplotlibrary

cfitsiomodule

cfitsiolibrary

XSPECmodule

XSPECmodels

S−Lang Interpreter

LibraryS−Lang

Off−The−Shelf (OTS) and System Libraries

CollectionModule

HYDRA Command Interpreter

modulePVM

PVMtoolkit Library

Scient.GNU

volviewmodule

GTKmodule

HDF5module

GSLmodule

HDF5library

GIMPToolkitLibraries

volpackrendering

library

module

PySLmodule

InterpreterPython

Figure 6: HYDRA Component Architecture

routinely convert IDL and MatLab scripts to S-Lang with little effort. Mapping such scripts tonon-vectorized languages would be considerablymore difficult.

8.3 Polyglot

Woven into the arguments above are notions thatno single language is best suited for every aspectof a modern analysis system, and that no sin-gle team will possess the skills, experience, andtime required to build a system that will be allthings to all comers. Venerable algorithms maybe coded in FORTRAN, and more recent effortsin C/C++, Python, or something else entirely.In principle this ought to be irrelevant. From theperspective of scientific analysis, the languageof implementation is of lesser importance thanwhether the functionality may be utilized withina given system, with robustness and accuracy.

HYDRA thus aims to enable science by es-chewing parochialism, again by building uponthe modular extensibility of ISIS and S-Lang.For example, ISIS already provides a modulewhich wraps numerous XSPEC models, most ofwhich are coded in FORTRAN. When colleaguesrequired specialized numerical routines for theiranalysis we did not code them ourselves, butrather provided S-Lang access to the GNU Scien-tific Library, again vectorized where appropriate,via yet another importable module.

While ISIS was not targeted at parallel ordistributed computation, we’ve employed it forsuch by wrapping the Parallel Virtual Machine

(PVM) within a module. Though the primarymeans of controlling ISIS interactively is throughcommand line input, we’ve extended it to therealms of graphical user interfaces and 3D visu-alization by employing the SLIRP code genera-tor (Noble, 2004a) to create Gtk, OpenGL, andVolpack rendering library modules.

Finally, we note the work underway in othergroups, particularly within the Python commu-nity (e.g., Loredo, 2001). Our first hope is thatcontributions from these efforts, at least at thealgorithmic level, are coded in compilable lan-guages so as to foster the widest possible reuse.Recognizing that this will not always be feasible,however, we envision mechanisms for invokingmethods in modules written for other scriptingenvironments and vice-versa. While early resultson this front (e.g. footnote 8) are encouraging,another aim is to establish stronger linkages be-tween S-Lang, Tcl, and Python.

8.4 Large Scale Computations

Aspects of this proposal, especially when com-bined with the increasing data volumes gener-ated by the telescopes of today and tomorrow,imply computing power well beyond the individ-ual workstation. When individual researchersconfront the limits of this approach, as we didwhen computing temperature maps, they arefaced with the prospect of assimilating tech-niques that have taken practitioners beyond ourniche years or decades to develop. Another aimof this proposal, then, is to identify ways of

13

Page 18: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

responding to this growing dilemma that aremeaningful and accessible to scientists steepedin the single-desktop model of analysis.

Our work distributing ISIS computations withPVM is a significant step in this direction. Itssuccess is evident in that more than a half- dozenconference or journal papers – from within ourinstitution and beyond – have already publishedresults derived in part from use of the PVM mod-ule. In fact, over the past year a significant frac-tion of the aggregate CPU cycles available on ourlocal network has been consumed by PVM jobsto compute a variety of CPU-intensive tasks in-cluding temperature maps, confidence contours,and photoionization codes. We expect this frac-tion to only increase as our techniques becomemore generally useful for a wider range of prob-lems. Clearly, this heavy usage of the local net-work negatively impacts our ability to performCPU-intensive experiments and testing relatedto HYDRA development. This impact is the pri-mary justification for our request of a Beowulfcluster in order to ensure dedicated computingresources are available.

9 Management Plan

9.1 Development Milestones

In the previous AISRP cycle (NRA 03-OSS-01),we were awarded an initial year of limited fund-ing to refine the HYDRA design and concept, aswell as address various issues and concerns. Thecurrent proposal seeks to extend that funding forthe full term and implement a number of specificdevelopment goals. Roughly, the HYDRA pack-age can be divided into three main developmentareas: source models, instrument models, andanalysis tools. The latter category encompassesthe filtering, fitting, and visualization discussedpreviously. Our current design calls for severalkey developments in each of these areas.

The expanded treatment of source models isone of the most unique aspects of the HY-DRA project. During the first year, we planto continue development of the generalized 3D

grid model prototype discussed in §4. Interimgoals include finalizing and documenting themodel interfaces, extending the number of sim-ple distributions currently treated, and optimiz-ing the model for distributed computation overnetworks. Development during subsequent yearswill focus on support for external models such as“snapshots” from adaptive-mesh hydrodynamicsimulations and Monte-Carlo raytrace simula-tions. We will also research possible ways toinclude more accurate radiative transfer effectsin computationally efficient ways.

A number of current and planned missions aregood candidates for HYDRA instrument mod-els including: Chandra, XMM, Astro-E2, andConstellation-X. During this first year, we willrestrict ourselves to implementing a fully func-tioning Chandra simulation module. We alreadyhave much of the infrastructure in place for sucha module from our MARX and ARDLIB de-velopment efforts. Our second priority will bethe incorporation of a module to support the re-cently launched Astro-E2 mission. The abilityto perform joint fits using high-spatial resolu-tion Chandra data and high-spectral resolutionAstro-E2 data is of particular interest. Subse-quent development will include models for theXMM RGS, Con-X, and ultimately instrumentsoutside the X-ray band.

For the first year of development, we have se-lected fairly modest goals for the filtering andfitting capabilities of the prototype. At a min-imum, we will replicate the standard spectralfitting capabilities users are familiar with frompackages such as XSPEC and ISIS. We plan todefine a minimal set of standard filters which po-tential users might employ such as: extracting aspectrum, a surface brightness profile, or a lightcurve. These filters can be used individually orin combination. Even this reduced functional-ity represents an advance over existing analy-sis packages. The more ambitious and flexiblefiltering described in §7 will be deferred to thesecond and third year of development where wewill prototype true multi-dimensional fits firstin a “binned” mode and then ultimately in a

14

Page 19: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

true “event-based” mode (e.g., as will be donein GLAST data analysis)

In terms of visualization, during the first yearwe will support volume animations, as well as in-teractive controls for manipulating and probingboth volumes and images. By the second year,we aim to employ these to tune modeling andfitting, as well as extend VWhere to include an-alytic functions as filtering constraints, support1D and 3D inputs, and create higher-level wrap-pers for file I/O and recording filter history. Bythe third year we’d like to establish the feasibil-ity of specifying a volume algebra to manipulateindividual objects within a volume space and/orspecify 3D models.

From the point of view of scriptability andhigh-performance computation, we’d like to be-gin by constructing high-level hooks allowingPVM to be utilized transparently for commonanalytical tasks. In the second and third years,we plan to establish bridges from S-Lang to Tcland IDL, so that the large DS9, XSPEC, andIDL user base may utilize our work directly.

9.2 Investigator Roles

This project will be conducted by a small, expe-rienced team of researchers at the MIT Kavli In-stitute. The investigators have worked togetherclosely for almost ten years developing a num-ber of software projects including: the Chandraobservatory simulator MARX, the ISIS spectralanalysis package, and portions of the CIAO X-ray data analysis system.

The Principal Investigator, Dr. Michael Wise,will be responsible for the successful completionof the project, and for the allocation of projectfunds. He will coordinate the work of the co-investigators, and assume final responsibility fortechnical decisions. His progress will be moni-tored by Dr. Jacqueline Hewitt, the Director ofthe MIT Kavli Institute. The PI is supported bysix Co-investigators:

Michael Noble, Co-I, will be responsible for theoverall system architecture and package integra-tion. He will also take responsibility for visual-

ization and consult on the API design and dis-tributed processing.

Dr. John Houck, Co-I, will take responsibilityfor the source model designs and fitting capabil-ities, as well as assist with API development anddistributed processing.

Dr. John Davis, Co-I, will take responsibility forthe instrument module design and assist with theAPI development and distributed processing.

Dr. Michael Nowak, Co-I, will be responsiblefor the filtering and selection design, as well asadvanced statistical analysis capabilities.

Dr. Dan Dewey, Co-I, will consult on sourcemodel designs, instrument simulation modules,and 3D visualization.

Dr. Claude Canizares, Co-I, will provide overallmanagement assistance and contribute to sourcemodel and instrument module specifications.

Partial support is requested for the two primarydevelopers (Noble and Houck) as well as the PI(Wise). The remaining Co-I’s will contribute tothe project as part of their scientific researchtime and no additional support is requested.

9.3 Equipment

As discussed throughout the text, we are target-ing both the sequential and distributed/parallelmodels of computation; all code developmentwill reflect these goals from the outset. In thefirst year we will purchase two multi-processorworkstations for initial development and test-ing. During the second year, we hope to obtainand install a Beowulf cluster, to provide a ded-icated computational resource for this projectand avoid the increasing congestion in our localnetwork (see §8.4). A 16-node cluster is equiva-lent in cost to only 3 or 4 high-end workstations,yet will help us avoid the looming bottleneck ofCPU saturation while providing the flexibility tosupport more communication-intensive compu-tations, e.g. implemented in the MPI (MessagePassing Interface). No additional computing re-sources are requested for year three.

15

Page 20: HYDRA: A New Paradigm for Astrophysical Modeling ...HYDRA: A New Paradigm for Astrophysical Modeling, Simulation, and Analysis Michael W. Wise, Michael S. Noble, John C. Houck, Michael

References

Arnaud, K. A. (1996). XSPEC: The first ten years. In Jacoby, J. H. and Barnes, J., editors, AstronomicalData Analysis Software and Systems V, number 101 in Astron. Soc. Pacific, Conf. Ser., page 17, SanFrancisco. Astron. Soc. Pacific.

Bagley, D. (2003). “The Great Win32 Computer Lang Shootout”, http://dada.perl.it/shootout/matrix.html.Connors, A. (2003). Power from understanding the shape of measurement: Progress in bayesian inference

for astrophysics. In Feigelson and Babu (2003), pages 19–36.Davis, J. E. (2000). A Framework for the Development of Multi-Mission Software. In ASP Conf. Ser. 216:

Astronomical Data Analysis Software and Systems IX, pages 607–+.Davis, J. E. (2001). The Formal Underpinnings of the Response Functions Used in X-Ray Spectral Analysis.

ApJ, 548:1010–1019.Feigelson, E. D. and Babu, G. J., editors (2003). Statistical Challenges in Astronomy, New York. Springer.Freeman, P. E., Kashyap, V., Rosner, R., and Lamb, D. Q. (2003). The statistical challenges of wavelet-based

source detection. In Feigelson and Babu (2003), pages 365–386.Geist, A., Beguelin, A., Dongarra, J., Jiang, W., Manchek, R., and Sunderam, V. (1994). PVM: Parallel

Virtual Machine, A User’s Guide and Tutorial for Networked Parallel Computing. MIT Press, Cambridge,MA ISBN 0-262-57108-0.

Houck, J. C. and Denicola, L. A. (2000). ISIS: An Interactive Spectral Interpretation System for HighResolution X-Ray Spectroscopy. In ASP Conf. Ser. 216: Astronomical Data Analysis Software andSystems IX, pages 591–+.

Hydra web site (2005). http://space.mit.edu/HYDRA.Lacroute, P. and Levoy, M. (1994). Fast volume rendering using a shear-warp factorization of the viewing

transformation. In SIGGRAPH ’94: Proceedings of the 21st annual conference on Computer graphics andinteractive techniques, pages 451–458, New York, NY, USA. ACM Press.

Loredo, T. (2001). “Astrostatistical Tools in Python”, AISRP, NRA-01-OSS-01.Meißner, M., Huang, J., Bartz, D., Mueller, K., and Crawfis, R. (2000). A practical evaluation of pop-

ular volume rendering algorithms. In VVS ’00: Proceedings of the 2000 IEEE symposium on Volumevisualization, pages 81–90, New York, NY, USA. ACM Press.

Noble, M. S. (2003). Rapid Development for Distributed Computing, with Implications for the VirtualObservatory. In ASP Conf. Ser. 295: Astronomical Data Analysis Software and Systems XII, pages 3–+.

Noble, M. S. (2004a). http://space.mit.edu/ mnoble/slirp.Noble, M. S. (2004b). VWhere: A Visual, Extensible ‘where’ Command. ArXiv Astrophysics e-prints.Ousterhout, J. (1998). “Scripting: Higher Level Programming for the 21st Century”, IEEE Computer,

March 1998.Peterson, J., Kahn, S., Paerels, F., Kaastra, J., Tamura, T., Bleeker, J., Ferrigno, C., and Jernigan, J.

(2003). High-resolution x-ray spectroscopic constraints on cooling-flow models for clusters of galaxies.ApJ, 590:207–224.

Peterson, J., Paerels, F., Kaastra, J., Arnaud, M., Reiprich, T., Fabian, A., Mushotzky, R., Jernigan, J.,and Sakelliou, I. (2001). X-ray imagin spectroscopy of Abell 1835. A&A, 365:L104–L109.

Peterson, J. R., Jernigan, J. G., and Kahn, S. M. (2002). Monte Carlo Methods for X-ray Dispersive Spec-trometers. In Astronomical Data Analysis II. Edited by Starck, Jean-Luc; Murtagh, Fionn D. Proceedingsof the SPIE, Volume 4847, pp. 195-206 (2002)., pages 195–206.

Scargle, J. D. (2003). Advanced tools for astronomical time series and image analysis. In Feigelson andBabu (2003), pages 293–302.

Scargle, J. D., Norris, J., Jackson, B., Hsu, T. M., Alt, A., Arabhi, S., Barnes, D., Gioumousis, P., Gwin, E.,Paungkaew, Sangrtrakulcharoen, Tam, L., and Tsai, T. T. (2004). Studies in astronomical times seriesanalysis, VI. optimum partition of the interval: Bayesian blocks, histograms and triggers. in preparation.

Wise, M. and Houck, J. (2005). Mapping Heating in Clusters of Galaxies. In 35th COSPAR ScientificAssembly, pages 3997–+.

Young, A. J. and Reynolds, C. S. (2000). Iron line reverberation mapping with Constellation-X. ApJ,529:101–108.

16